Moving from manual testing to automated testing is a big task. Like any major software project, there are a lot of unknown pitfalls that might trip up your implementation. As a leader in your organization, it’s important that your team’s work deliver a good return on investment (ROI). The truth is that it’s difficult to measure ROI on software testing. And measuring test automation ROI is even more difficult.
The reason it’s difficult to measure test automation ROI is because there are a lot of variables at play. If you’ve been shipping software for a while, you’re probably pretty comfortable with your current testing process. Why upend the apple cart and possibly wind up in a worse state. How much would automating your tests really save you?
The truth is, I can’t answer that last question for you. There are many arguments in favor of an automated testing platform. I can tell you that adopting one will likely save you money. I can’t tell you how much. But I can tell you how to measure how much you’ll save. In this post, we’re going to walk through the things you need to measure when you adopt a test automation platform. We’ll talk about how those variables interact, and how you can use them to determine how much test automation saves you.
Calculating ROI on Test Automation
Software leadership teams make a common mistake while testing test automation ROI. In short: they’re too naive. They recognize testing as an important factor in their software’s total cost of ownership. However, they treat software testing purely as a cost center. You put money into the QA department, and hopefully, you get better software as a result. My experience is that this attitude is more likely to be found at larger, more established companies. Many smaller companies just skip QA altogether, which is a decision with often-disastrous results.
When you view QA as a cost center, your ROI is necessarily going to be limited to cutting costs. If QA doesn’t provide value, but simply prevents errors, how can investing in them ever provide more value?
As we’ve noted, this is a naive view. However, we’ll start from there with our calculations, then layer on some additional information that we can use to provide better ROI measurement.
The Basic Measurement
The most basic way of calculating ROI on test automation is to measure how much QA time it saves. For instance, if it costs 500 hours to write a bunch of automated tests, but those 500 tests reduce the quantity of manual testing performed each week by 20 hours, it’s a simple calculation. We’ll be neutral on our time investment in 25 weeks, and if we assume developers are twice as expensive as QA engineers, we’ll be saving money in a year.
My experience is that people like calculations like this because they’re easy to understand. Unfortunately, they don’t provide great insight into your company’s true costs. Sure, you can layer the monthly cost of the software you’re using to automate in there. That’ll make your calculation a little better. But that’s just not enough to get you to a good measurement.
So instead, we’ll talk about some additional things to measure in order to get the best data you can for calculating your team’s ROI.
Tests Have to Be Maintained
Perhaps the most significant piece that’s missed in our basic calculation is that you can’t write tests once and leave them forever. Tests are software, just like any other bit of code. This means that your ROI for testing needs to take into account the ways in which those tests will change over time. Some of that change is going to come from adding new features. In our previous example, we had 500 tests. Those 500 tests are just the start. If at the end of your first year of test automation, you still have just 500 tests, you’re doing something wrong. Instead, the number of tests in your test suite will grow as you add new features.
However, it’s not just new tests. Many features change the way that something works. This means that developers spend time modifying existing tests to suit new code they’ve written. This is time that you need to account for when calculating your ROI.
Measuring this time is challenging. Developers are notorious for having difficulty measuring how much time it takes them to test something. What’s more, they’re resistant to attempts to better measure that time. In order to best calculate your ROI for test automation, you need to work with your devs to understand as best you can how much time they’re spending writing and maintaining tests. These measurements are critical to your cost calculations.
QA People Often Aren’t the Best Test Writers
This is another common mistake I see out of companies transitioning to test automation. They think that because they have QA specialists, those specialists can translate test plans to code. Many QA specialists are not equipped to do this, and not interested in learning. That’s OK! Moving to automated testing doesn’t mean you’ll replace your entire QA department. Some of your QA specialists are likely interested in learning how to code. Mentoring these people and helping them bridge that gap by automating testing is a great idea. If you have enough people like this, that mentoring time is another cost you need to account for. As they’re learning, they’ll be spending less time manually testing things.
If you don’t have those people, you’ll need to hire them. That’s a cost you need to account for. If you’re going to rely on more automation, you need people who are well-versed in maximizing both software test coverage and who are good at testing software from a variety of directions. The key to successful testing automation is having good tests. Make sure your ROI calculations are factoring in the cost of finding or training people to write good tests.
QA People Make Mistakes
How many times have your QA team signed off on a release, only to discover bugs in production? If it’s more than once, you know this pain. I don’t bring this up to criticize QA teams. They’re a critical part of the software development life cycle. However, QA specialists still make mistakes. Things slip through the cracks. QA specialists test the same software every day, in the same ways, for weeks or months. That kind of monotony adds up, and they sometimes miss things. That’s the nature of human beings.
If you’re estimating the cost of releasing software bugs, QA missing things needs to be factored into your ROI calculations. The benefit here is that automated software never skips a step. It never assumes something just works, because it worked last time. It tests everything, every time. This means fewer things slip through the cracks. This is one way that QA and testing automation provide real value to companies. They’re not just cost centers.
Test Automation Works at a Scale Humans Can’t
Scaling your testing is something on the minds of all software teams right now. In the past few weeks and months, millions of people have transitioned to working online due to the novel coronavirus COVID-19. Millions more people started using tools like Slack and Zoom to communicate than those companies anticipated. Thankfully, those tools mostly stood up to the stress, because their teams tested them at scale.
You can’t hire an extra million QA specialists to test your software under load. But you can use a testing automation framework to do that. By leveraging the power of testing automation, you can anticipate your scaling needs and identify problems before they become outages. Factoring these savings into your ROI calculations is another way that QA provides value to your company. Test automation both provides visibility into your needs as your company grows, as well as helps you anticipate what a sudden demand spike would mean for your software.
Unlocking Your ROI Through Better Software
We’ve talked about a few different ways to build better ROI calculations for your testing automation push. While it’s all well and good to have great calculations, your goal is still to maximize that ROI. This is where a product like Testim’s Test Automation Tool comes in. The best way to maximize your ROI is always going to be minimizing the time spent writing and designing tests. That’s what Testim is here to help with.
If you’re thinking about test automation, but you’re concerned about maximizing your ROI, Testim can help you discover ways to cut that initial cost. You’ll minimize your long-term costs, and maximize the quality of your testing architecture to meet the needs of today and tomorrow. Contact Testim and find out how you can maximize your team’s testing ROI today.
This post was written by Eric Boersma. Eric is a software developer and development manager who’s done everything from IT security in pharmaceuticals to writing intelligence software for the US government to building international development teams for non-profits. He loves to talk about the things he’s learned along the way, and he enjoys listening to and learning from others as well.
Download Testim’s free ROI tool.
