Outbrain is a leading global native advertising platform. You may not have heard of Outbrain, but you’ve undoubtedly used their service. Outbrain helps over one billion people discover content, products, and services that may be of interest to them. They specialize in native advertising—serving written content and videos that are incorporated into the platform on which it is displayed.

If you’ve visited news sites like CNN, BBC, or LeMonde and read an article, you’ve likely seen related articles or advertisements that were curated by Outbrain. Outbrain helps connect publishers and advertisers with people who wish to engage with their content.

We spoke with Tsachi Shushan, head of front-end guild, about their use of Testim.

Challenges

Outbrain’s software is critical to its success, connecting content publishers with content consumers. The Outbrain native content engine provides users with interesting and relevant content on premium publisher’s sites. Testing is a challenge because the application manifests as feeds on over 100,000 websites that they don’t own. Yet, Outbrain’s success depends on readers clicking on the content displayed in these feeds and getting the proper outcome—taking readers to the right place and recording the transaction.

Outbrain is a modern development company in a microservices environment with a continuous delivery pipeline all the way through production. They release code to production several times a day and rely heavily on tests. “We use unit tests, integration tests, and full end-to-end integration tests. The relevant tests run on every change to the related code,” said Tsachi. He added, “the web application testing was always the most painful part.” They were using Selenium and Protractor, but not every team could keep up. Tsachi stated,

Many teams wanted to do end-to-end tests, but not every team had an automation engineer who could support them.

Even the teams with Automation Engineers (AE) were having difficulties. Tests took a long time to author, they failed often and took a lot of time to debug. Even the runs were configured differently for each team. The tests took a long time to run and were unstable. Tsachi estimated that they were spending 10-15% of their Web engineering hours on automation. For an organization of over 50 engineers working on web applications, that is an equivalence of five to seven full-time engineers. They were getting feedback from the team that they were spending too much time automation tasks. They needed a better solution.

They knew that as their application grew they would want to spend more time on regression tests and the current model with Selenium and Protractor couldn’t effectively scale.

Turning the page

Outbtrain did a traditional POC evaluation process that lasted around a month comparing many solutions including TestCafe, TestCraft, Cypress, Endtest, Ghost Inspector and more. Testim came out on top as the best fit.

Tsachi said that their criteria included ease of use, the ability to quickly author tests, the completeness of functions and the responsiveness of the support team. The product was easy to use and also worked well for their application. Tsachi emphasized,

The recorder was at least one level above every other solution we tried. It worked in every scenario including drag and drop, iframes, mouse gestures and more.

He was also impressed by the stability provided by the Smart Locators. “They made the tests very robust. They didn’t fail on minor code changes. It’s pretty unique among the products we tested,” he added. Finally, they also looked at the support experience. Tsachi said, “the Testim support team was very responsive and helpful.”

Solution

Outbrain chose Testim for end-to-end functional UI testing for their applications. In total there are about 20 applications that they test with Testim including the feeds embedded in over 100,000  websites across the world. Their applications include their content advertising platform that allows customers to create campaigns as well as the publisher application that helps publishers manage content and settings for the Outbrain feed.

Outbrain has incorporated Testim into their CI pipelines and processes. This includes running functional tests on new feature commits and regression tests on larger releases, all triggered from a Jenkins CI build. Outbrain runs their Testim created tests on their own Selenium grid.

One of the goals of the test automation project was to get as many teams as possible involved in end-to-end testing. Before Testim, there were only six teams doing end-to-end testing with Selenium and Protractor and now there are 14 teams involved, largely due to Testim’s ease of use.

Developers are able to switch context more easily which makes writing tests more acceptable. They didn’t want to write coded tests, and Testim allows them to record and configure tests. “Before Testim, new tests in a large application were taking one to two days to write and make stable. The same tests in Testim take between 20 and 30 minutes,” said Tsachi.

Tsachi talked about the overall change, “we’ve received lots of good feedback about the process and writing and fixing tests. The developers like that they don’t have to write code, commit code, wait for code review, etc. Instead,

Developers can just record tests, it makes the context switching for developers much easier.

Benefits achieved

• Grew number of teams using E2E tests from 6 to 14, a 233% increase in six months

• Reduced the time spent automating tests from 10-15% of Web engineering team’s time to 5% (2-3X reduction)

• Increased test coverage on the Testim projects by an estimated 50% resulting in higher quality

• Reduced time to write stable tests from 1-2 days to 20-30 minutes, a savings of over 95%

• Trust has increased because developers know they have a safety net that works