Connecting Tests to Your CI: The Good, the Bad, and the Ugly

We often hear the software quality success stories about connecting your test automation solutions to your CI, but how does this impact your software quality and testing practices?

Scaling your test automation is never easy and sometimes this means facing honest facts, while continuously delivering results for your customers. But this doesn’t mean that you and your software development teams need to recreate a Western film showdown during your sprints and retros.

We asked our test automation leaders in the Testim Community to share their candid experiences to help you on your journey for continuous velocity and quality improvement. Read on for pro tips from our test automation industry leaders below:

The Good:

• The good? Connecting test automation solutions to your CI enhanced quality processes is always a good thing for your customers. During the CI/CD process, it doubles your software quality efforts to make sure that all your automation suites run successfully. If something breaks, then you have to fix it. So automatically, these breakdowns really enhance your overall testing and quality for deliverability.
• As Martin Fowler said, “Continuous Integration doesn’t get rid of bugs, but it does make them dramatically easier to find and remove.” Integrating automated tests can help you detect errors quickly by which your team can move fast while keeping high-quality standards.
• Automating your tests adds the goodness of a faster feedback loop, reduces risk, and eliminates time wasted in the software development lifecycle.
• Communication and synchronization between teams hold the key here. Otherwise, in no time, this setup could get bad.
• More test visibility helps your teams more quickly find issues with the tests themselves or with code.
• Connecting to your CI makes your test visible to the entire development team, easier reporting when you fully integrate logs into your tests and catches errors in logging. This can only help you, as a test automation leader, improve channels in communication with the development team!
• Velocity means moving with confidence, which results in quality. Remember that fast feedback means shorter cycle time!

The Bad:

• If the outcome is not generating any incremental value proposition it can’t get uglier than just bad.
• There’s a cause and effect that’s inherent in visibility, which really starts to tie the actions we take as a team to the results of the tests. Spoiler alert: for your teams with low psychological safety, as a leader, you can get blamed for messing up the release or worse, this can create situations whereeammates may try to hide results when something breaks. Yikes.
• At times when tests fail one after the other in your automation solution/suite then you have to get ready for enabling your tests to run again effectively without impacting other test cases.
• Connecting to your CI means often messy inconsistencies with running tests on local vs running on CI.
• Not doing trunk-based development reduces the benefits.
• False negatives (i.e. timeout or spurious test issue) can reduce your confidence in tests.
• When your tests are failing remotely and not locally or when all your pipelines got canceled. You might find yourself having zero test runs in the morning.

The Ugly:

• Connecting to your CI feels like it really starts to show any flaws in your test methodology. Are you writing so many tests that it eats up all your Github actions minute>. Are your tests so light that they’re not really telling you about your system? Is everyone involved in writing tests and understanding the information they return?
• Watch out for false alerts and that add to your team’s miseries.
• The period of frustration during the maturity phase to stabilize CI/CD
• Poorly designed tests & test code, unnecessary tests, flaky tests, lack of good patterns and practices, can make teams go even slower than they went without test automation.
• Timing issues can definitely contribute to the ugly factor. For example, sometimes the first tests run after a deploy fails. It’s not ideal to have to play (read guess) with timing. When you see in the morning that the device farm you are using (set up by error your devices in Chinese( and a lot of UI tests failed). Lesson learned here.
• Aimlessly integrating scripts onto CI/ CD leads to the ugly.

Connecting your automation solutions to your CI doesn’t have to be a sunset duel with your software development teams. Remember that connecting to your CI doesn’t have to be ugly as it will always help you give a more quality-driven product that your customers and key stakeholders will love.

As Savanna Goodman, automation advocate at IntSights said in her Testim Community leadership session, “it’s important to have a framework based on trust, flexibility, and visibility to help you scale and influence test automation across your teams.” It is imperative to find a test automation solution that helps you scale testing with control, management, and insights. We know that our latest innovation, TestOps will help bring the good out of your experiences in connecting to your CI. Learn more here.

Join the Testim Community
If you are interested in connecting with industry leaders, attending live community events, or networking with other developers and QA automation professionals, join us in the Testim Community.