Does your development team spend too much time waiting on their test suite to run? Do they constantly rerun the test suite after failing tests because “rerunning magically fixes it”? If your developers have these problems, there’s a good chance their test suite doesn’t follow the test automation pyramid.
The test automation pyramid is an important concept that all software developers should be familiar with. It’s a framework that can help guide the development team into producing a higher-quality product. The test pyramid also reduces the time it takes developers to find out if they introduced a breaking change. As well, it can lead to a more reliable test suite.
This post is going to dive into the test automation pyramid and introduce some strategies for managing it.
Table of Contents
- Test Automation
- What’s the Test Automation Pyramid?
- Unit Tests
- Integration Tests
- End-to-End Tests
- Conclusion
Test Automation
Before we dive into the test automation pyramid, let’s define test automation. I’m not going to reinvent the wheel, so I’m going to borrow the definition from this blog post: “Test automation is the practice of running tests automatically, managing test data, and utilizing results to improve software quality. It’s primarily a quality assurance measure, but its activities involve the commitment of the entire software production team.”
Test automation can reduce the amount of time developers spend testing their software because the environment can be prepared and cleaned up automatically. Not only that, but it can report the results of the test run to be used for future insights. This can help developers debug problems faster.
We live in a world where you have new updates going out almost every day. The practice of Continuous Integration and Continuous Delivery has taken over the world as it has proven to be beneficial to the business. And since testing is a crucial part of this process, it’s also important to have an efficient testing process. Test automation plays an important role in this as it reduces the time spent on testing and improves accuracy.
So now that we know what test automation is, let’s see where the pyramid fits in.
What’s the Test Automation Pyramid?
The test automation pyramid represents the various types of tests and the frequency at which they should appear in the codebase’s test suite. It’s all about giving the developer immediate feedback that code changes don’t break existing features. The test pyramid has three distinct sections. Unit tests are at the lowest level, integration tests make up the middle tier, and end-to-end tests form the top.
The test pyramid helps developers deliver quality software.
Why do we need to make a distinction between the types of tests in the codebase? Let’s answer this question by breaking down the tiers. I’ll also define some common tooling in each of the levels and introduce some tips for managing the test automation pyramid.
Unit Tests
What are Unit Tests?
Our test pyramid needs a strong base, and that’s where unit tests come in. This means they’re the most common type of test in the test pyramid.
What’s a unit test? A unit test focuses on testing a very small component or piece of functionality of the codebase. Its goal is to validate that the unit behaves as expected in isolated conditions. This could be an individual function or a class. Developers should focus on testing a number of scenarios ranging from happy path to error handling.
Toolset
Mocking frameworks, such as Java’s Mockito or JavaScript’s Sinon, are common tools found in the unit test tier. Mocks give developers the ability to control the behavior of components used by the subject under testing. For example, a class that depends on communicating to a RESTful service may use an HTTP client library. The developer may have to mock out the HTTP client in order to control the types of responses returned by the web service.
Considerations
The unit test suite should run quickly because they’re the largest subset of tests in our pyramid and will continue to grow in number as new features are added. The unit test suite should be run any time a developer makes a code change. This gives the developer immediate feedback on whether or not their new code broke anything in the codebase. A fast unit test suite encourages developers to run the suite more often, which cuts down on the time spent debugging problems.
One way to build a strong unit test suite is through the practice of test-driven development (TDD). TDD can naturally lead to a strong unit test suite because it requires a test to be written before any code. As new features are added, new unit tests are created.
Unit tests are great for testing small pieces of a codebase. However, unit tests aren’t representative of a deployed application. And they don’t test the interactions of the app with the outside world. This is where integration tests come in.
Integration Tests
What are Integration Tests?
Unit tests aren’t enough to ensure the quality of a codebase, so now we’ll introduce integration tests. Integration tests make up the middle tier of our test automation pyramid. This means that integration tests should not appear as often as unit tests. Integration testing can be an overloaded term in the software development world, which is prone to confusion. I’ll define integration tests as tests that validate interactions with external components. Let’s unpack that a bit. External components can be anything that lives outside of your application, but that your app still depends on.
For instance, databases are a very common external dependency in software applications. It’s important to test that your application interacts with your database as intended. Database integration tests should ideally run against a real instance of a database similar to the production environment. However, this requires managing the database locally or in the build pipeline. Docker can make this easy to manage because it offers images of many of the popular databases including MySQL and even Microsoft SQL Server.
Furthermore, the integration with external services should also be tested. These types of tests can also be called service tests. Service integration tests should focus on the interaction of your code with the external service such as a RESTful API.
So why do developers even bother with unit tests if integration tests show our code is functioning with the external dependencies?
Why do we need Integration Tests?
Some dependencies are outside our app, and the feature we’re testing might require to communicate with these dependencies. This will make a call to the database or web service, which can cause the integration test to run slower compared to unit tests. Furthermore, if we’re testing an integration with an external web service, it’ll require a preproduction environment capable of testing against. Finally, it may be difficult to test each scenario from an integration point of view, such as error cases, because we cannot control the responses from the dependency.
End-to-End Tests
Finally, we’re at the top of the test automation pyramid.
What are End-to-End Tests?
At the end of the day, we want to make sure our entire application is functioning as expected, and that’s exactly what end-to-end tests help with. End-to-end tests are what they sound like; they test that your application is working from start to finish. In other words, they test the integration of the front end with the back end.
I like to think of end-to-end tests from the perspective of the end user. How would a person interact with my app? And how can I write a test from the user’s perspective? Let’s dig into an example. Our application more than likely has a login page. And we may want to test that the login page is working. This would require the following steps:
- Find the username text field and type the username.
- Locate the password text field and type the password.
- Click the submit button.
This could be a very time-consuming and tedious process for developers to test their code change each and every time. Not only that, but different developers may have different strategies for this manual test. So how can we make this process easier to manage and more reproducible?
End-to-End Testing Frameworks
Selenium is a testing framework that automates interactions within a browser. Selenium provides a domain-specific language that allows developers to write tests that interact with a web app running inside the browser. We could write a test with Selenium to follow the steps of the login process. This also shows that the integration with the UI to the back-end API is working together. If you’re looking for a codeless or hybrid solution, you can try tools like Testim.
End-to-end tests are at the top of the test automation pyramid because they can be slow and very fragile. And like integration tests, they may depend on external dependencies that may be unreliable.
Important of Test Pyramid
We’ve come a long with the complexity of applications. If you look at any typical modern application, you’d find multiple features, API usage, and a lot more. With this increase in complexity, the complexity of testing applications has also increased. Additionally, there’s another catch – you need the testing process to be fast so you don’t delay deployments.
There are multiple factors you need to think of in manual testing. Manual testing takes more time than automated testing, it’s less accurate and more prone to human error. You also need to test the application after every update. And you might need to test the same feature multiple times. It’s a human tendency to get bored of repetitive tasks. And boredom can lead to ignorance or negligence and this might lead to mistakes.
To deal with all these bottlenecks, you need a process that is efficient, and test pyramid is exactly this. Test automation is like adding a sports engine to your car and test pyramid is like adding extra turbo to it. Test Pyramid acts as a guideline to get better results from testing with minimal effort. As a result, it tremendously improves the testing process.
Best Practices
We’ve been talking about test pyramid, its benefits, and how it makes the testing process efficient. But is following the test pyramid approach enough to reach the most efficient stage? Well, I’d say there’s always room for improvement. Test pyramid is effective, but you can do a lot more to improve testing. So, here are some best testing practices:
- Plan and decide what test cases to automate and the scope of the tests.
- Choose the right tool for your use case.
- Prioritize tests.
- Write clean test code.
- Prepare test cases and scenarios using good quality test data.
- Avoid test duplication.
- Put tests into your deployment pipeline.
- Consider exploratory testing so you don’t miss out on anything.
Conclusion
Let’s recap what we’ve learned now that we understand the concept of the test automation pyramid.
- The test automation pyramid is a framework that defines various types of tests and the number of times they should appear.
- Unit tests form the base of the test pyramid. They should be frequent, and they should run fast.
- Integration tests are the middle tier of the pyramid. These tests focus on interactions of your code with the outside world, such as databases and external services.
- End-to-end tests top the test pyramid. They’re written from the perspective of a user and should test that your entire application is functioning from front end to back end.
Now that you know what the test pyramid is, do you think your development team’s codebase has a strong pyramid? When a team has a robust test pyramid, coupled with test automation, it can help the development team ship features faster and with higher quality. Learn more about test automation and how to choose a test automation platform that fits your development teams’ needs.
This post was written by JT Wheeler. JT got his start working as an engineer in the manufacturing industry. From there, he moved on to become a developer at a software consulting firm. He’s written software that spans a number of industries ranging from industrial automation to inventory control to financial systems, and he’s passionate about learning and teaching new technologies and software practices.
