CDP mode with shadowroot interaction

[nikita-devbox]

Greetings,

I am happy to have discovered this well-maintained project and have been studying the docs now for a week and been conducting small tests.

During the past few days however, I've struggled with detecting / interacting with elements under one, or multiple open shadow root elements, which according to the example test files of seleniumbase repository should work, so I am forced to ask, if anyone could point out what am I doing wrong.

My issue as follows:

• In the below image you'll be able to see a semi-simple HTML structure with one shadow root at the top.
• I've tried to detect the button with:

self.selbase.cdp.find_elements(selector) // where selector = "onboarding-modal::shadow #w-button"`

• trying to click on it directly without finding it first:

self.selbase.cdp.gui_click_element(parent_selector)
try1: Element {onboarding-modal w-button[variant="secondary"]} was not found after 7 seconds!
try2: Element {#modal-div w-button[variant="secondary"]} was not found after 7 seconds!

My questions are:

• is it possible to detect and then click on the button element under open shadow root node?
• if yes, how? If not, how would you do it?
• is it possible to be done without disconnecting from CDP mode? If not and reconnection to the Webdriver is required, what does "safe way" to reconnect without detection mean in practice? The way I see it, it must mean that it has to be done while the connection to the website is closed?

(Note that reconnecting allows anti-bots to detect you, so only reconnect if it is safe to do so.)

Thank you in advance!

[mdmintz]

As in the example below, you have to use sb.cdp.get_nested_element(parent_selector, selector) in order to get an element that's nested inside Shadow root elements.

from seleniumbase import SB

with SB(uc=True) as sb:
    url = "https://seleniumbase.io/other/shadow_dom"
    sb.activate_cdp_mode(url)
    element = sb.cdp.get_nested_element("fancy-tabs", "section")
    print(element.text)

Once you have an element, you can use the CDP Element API to do more, such as element.click().

Related methods:

SeleniumBase/examples/cdp_mode/ReadMe.md

Lines 401 to 402 in 8317bc8

	sb.cdp.nested_click(parent_selector, selector)
	sb.cdp.get_nested_element(parent_selector, selector)

[nikita-devbox]

I used sb.cdp.get_nested_element(parent_selector, selector) to successfully to locate an element under an iframe.
It appears not to work with this particular shadow root configuration. Here are results from two of my test functions for troubleshooting:

self.selbase.cdp_find_nested_element("div[id="modal-div"]", "onboarding-modal[brand="TORI"]")
input("dev pause 1...")
self.selbase.cdp_find_nested_element("onboarding-modal[brand="TORI"]", "div[role="dialog"]")
input("dev pause 2...")

Console output:

(debug): get_nested_element(): searching (onboarding-modal[brand="TORI"]) in parent (div[id="modal-div"]) => located! => webdriver is in undetected mode
element= = <onboarding-modal silent="True" locale="fi" messages="{}" brand="TORI" clear_input="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6CF1F560>" click="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6CF1DB20>" flash="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6CF1DBC0>" focus="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6CF1DC60>" gui_click="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6CF1DD00>" highlight_overlay="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6CF1DDA0>" mouse_click="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6CF1DE40>" mouse_drag="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6CF1DEE0>" mouse_move="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6CF1DF80>" press_keys="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6CF1E020>" query_selector="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6CF1E0C0>" querySelector="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6CF1E0C0>" query_selector_all="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6CF1E160>" querySelectorAll="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6CF1E160>" remove_from_dom="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6CF1E200>" save_screenshot="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6CF1E2A0>" save_to_dom="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6CF1FD80>" scroll_into_view="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6CF1FE20>" select_option="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6CF1FEC0>" send_file="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6CF1FF60>" send_keys="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6D780040>" set_text="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6D7800E0>" set_value="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6D780180>" type="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6D780220>" get_position="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6D7802C0>" get_html="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6D780360>" get_js_attributes="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6D780400>" get_attribute="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6D7804A0>" get_parent="<function CDPMethods.__add_sync_methods.<locals>.<lambda> at 0x0000028B6D780540>"></onboarding-modal>
dev pause 1...

(debug): get_nested_element(): searching (div[role="dialog"]) in parent (onboarding-modal[brand="TORI"]) => located! => webdriver is in undetected mode
element= = None
dev pause 2...

[Rapid1898-code]

I tried the solution using the following code:

from seleniumbase import Driver

link = "https://www.yell.com/ucs/UcsSearchAction.do?keywords=accountants&location=newport-gwent&pageNum=1"

driver = Driver(uc=True, headless=False)
driver.uc_activate_cdp_mode(link) 
driver.cdp.get_nested_element("div#usercentrics-root", "button[data-testid='uc-accept-all-button']").click()
driver.quit()

But i still get an error - the element in the shadow root is not found:

(seleniumBase) C:\DEVNEU\Fiverr2025\PROGRAMS\Yell_UK>python test.py
Traceb'ack (most recent call last):
  File "C:\DEVNEU\Fiverr2025\PROGRAMS\Yell_UK\test.py", line 7, in <module>
    driver.cdp.get_nested_element("div#usercentrics-root", "button[data-testid='uc-accept-all-button']").click()
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'click'

[mdmintz]

sb.cdp.gui_press_keys("\t\t\t\t ") will press that button.

[Rapid1898-code]

I tried this with the following code:

from seleniumbase import Driver

link = "https://www.yell.com/ucs/UcsSearchAction.do?keywords=accountants&location=newport-gwent&pageNum=1"
driver = Driver(uc=True, headless=False)
driver.uc_activate_cdp_mode(link) 
driver.cdp.sleep(1) 
driver.cdp.gui_press_keys("\t\t\t\t ")
input("Press1")
driver.quit()

But its still not working - i get this screen after the program run

[mdmintz]

from seleniumbase import SB

with SB(uc=True, test=True, ad_block=True) as sb:
    url = "https://www.yell.com/ucs/UcsSearchAction.do?keywords=accountants&location=newport-gwent&pageNum=1"
    sb.activate_cdp_mode(url)
    sb.sleep(2)
    sb.uc_gui_click_captcha()
    sb.sleep(2)
    for i in range(8):
        sb.uc_gui_press_keys("\t")
        sb.sleep(0.5)
    sb.uc_gui_press_keys(" ")
    breakpoint()

Adjust timing as needed.
Websites may also change based on GeoLocation, so update as needed there.

[Rapid1898-code]

Hello - it seems get_nested_element is still not working - find attached this example -

from seleniumbase import Driver
link = "https://www.daynurseries.co.uk/daynursery.cfm/searchazref/65432200502"
driver = Driver(uc=True, headless=False)
driver.uc_activate_cdp_mode(link)
input("Press!")
driver.cdp.get_nested_element("div[class='needsclick']", "button[aria-label='Accept All']").click()
input("Press")

I get this error:

(seleniumbase) C:\DEV\Fiverr2025\TRY\carystt>python test.py
Press!
Traceback (most recent call last):
  File "C:\DEV\Fiverr2025\TRY\carystt\test.py", line 9, in <module>
    driver.cdp.get_nested_element("div[class='needsclick']", "button[aria-label='Accept All']").click()
AttributeError: 'NoneType' object has no attribute 'click'

Is the function "get_nested_element" still not working or am i doing something wrong?

[mdmintz]

Some open shadow-roots can't be penetrated via CDP. (Closed shadow-root can never be penetrated.)

You may have to use PyAutoGUI as a workaround. Maybe count the number of times you need to press TAB before clicking SPACE or ENTER to trigger the button. There are some examples of that, eg:

SeleniumBase/examples/raw_hobbit.py

Line 7 in 0694314

sb.uc_gui_press_keys("\t ")