How Testim.io handles Shadow DOM
Shadow DOM & Custom Elements – Background
If you are familiar with shadow DOM and custom elements, feel free to skip this intro, and continue reading from the next section, where I discuss the challenges of testing web pages that include Shadow DOM.
Like any other field in programming, web development has evolved while balancing between “code duplication” and “code isolation”. Following the design and the architecture of common frameworks, a custom elements standard was developed, alongside the shadow DOM specification. These two simple, yet powerful concepts, enable sharing and reusing components, while using the DOM (Document Object Model) as an interface for interacting between these components.
The DOM is essentially just a tree of elements, while Shadow DOM lets you place the children in a scoped subtree, called Shadow Tree, so document-level CSS can’t restyle a button by accident, for example. This isolated sub-tree could be then attached to any element:
Here is a simple example:
Notice that we used here “open” shadow DOM mode, which allows us to access the sub-tree.
Custom Elements enables wrapping such “DOM trees” with a custom tag name, while listening to attribute changes:
These two standards offer many advantages, starting from the basic use-case of using short and simple CSS selectors – without the fear of colliding class names, and on to the ability to the ability to build framework-agnostic components, which enable the sharing of components between teams or projects.
Further info about Shadow DOM and Custom Elements is available at MDN.
What problems can Shadow DOM create?
Along with the tremendous advantages offered by Shadow DOM and Custom Elements, there are also a few challenges, specifically when trying to perform automated UI testing on these elements:
The most common challenge is locating an element on which actions will be performed, which is a fundamental part of any UI test.
For instance, a typical mouse-click in an end-to-end test could look like this: browser.click(‘#some > css .selector’);
However, in case we want to click on a “Shadowed” element – there is no valid css, xpath or any other query selector to use. Another common challenge is that after we already found an element, we may want to make sure it is clickable and that there is nothing overlapping it. One way of achieving this is by running the following: document.elementFromPoint(x, y) === element
However, this would break as well, since document.elementFromPoint would return the “Shadow Root” of the “Shadowed” element.
These issues may remind you of Iframe-related issues. Although testing inside Iframes is not trivial, Iframes are different in that they are more isolated than Shadow DOM, since they have their own js execution-context, their own body and some other behaviors that allows the testing software to act as if Iframes were a totally separate webpage.
How to overcome these issues? Testim.io vs others solutions
In web testing there are three main alternatives to locate actions on elements:
- Using WebDriver protocol – one way is using the WebDriver protocol, which is based on HTTP and used mainly by Selenium.
- Executing the entire JS code – Alternatively, the entire js code can be executed directly on the tested web-page like in early versions of Selenium, which was changed in favor of the WebDriver protocol.
- Using DevTools protocol – A third path, which is implemented by Puppeteer, is using the DevTools protocol. Currently this solution supports only Chrome and Firefox browsers.
Each of these approaches have their own pros and cons, so at Testim we are using all of them in a hybrid fashion, depending on the specific use case and need.
So, let’s take a step back and understand what happens in each of these methods when our test runs the following: browser.click(‘#some > css .selector’);.
We will also show how we can use this method to click on a “Shadowed” element:
WebDriver protocol
In this case, clicking an element involves two HTTP requests:
First we find an element requesting POST /session/{session id}/element including a CSS selector in the request body.
The previous request returns a unique Selenium element ID, which we use to click the located element: POST /session/{session id}/element/{element id}/click
Obviously, a real test is more complex as it usually includes the initialization of the session and probably some more actions.
Therefore the strategies for locating an element are very limited and there is no way to use these requests to locate a “shadowed” element (although there are some proposals for solutions).
DevTools protocol
This implementation varies between browsers. Chrome passes clicks directly to the operating system by using coordinates, indicating where to click. This still doesn’t help us when clicking on Shadow DOM elements since we don’t have their coordinates without finding them first.
There is however an option to do that, but it is not fully supported by all browsers.
JavaScript implementation
Last but not least is implementing the entire flow in JavaScript.
This basic click: browser.click(‘#some > css .selector’); would be translated into the following code when running on a website:
const element = document.querySelector('#some > css .selector');
const event = new MouseEvent("click", {});
element.dispatchEvent(event);
To convert this code so that it supports clicking on a Shadowed element we want to click the first counter button on this example:
Doing this would require us to call multiple times, one for each document/shadow root:
let element;
element = document.querySelector('#counter1')
element = element.shadowRoot.querySelector('button');
const event = new MouseEvent("click", {});
element.dispatchEvent(event);
Some Selenium clients implemented a similar JavaScript solution (Java, JS). The downside is that each level of Shadow DOM nesting, would require a new query selector. In complex web systems with many levels of nesting, this approach can become very complicated.
The difference between Testim.io and other implementations is that when using Testim.io, this code is generated automatically by means of our smart locators, which dynamically create element selectors using AI. You can find further information about that in testim.io documentation.
Summary
Testing Shadow DOM is challenging due to its inherent isolation. While the mainstream testing frameworks currently don’t have an answer for these challenges, others provide various ways of solving these challenges, usually with the cost of complicated and therefore more fragile tests. Testim.io takes the entire mission of selecting elements and does it for you, including in the case of a “Shadowed” element.
P.S. In case you are interested, Testim released a free recorder for Puppeteer. Check it out and use it as often as you like.
