Unit Tests are in the house!
I’m a big advocate for TDD (Test Driven Development). Research shows TDD has a high correlation to quality code, as it prevents bugs from forming in the first place. Also, the cost of authoring tests and fixing bugs is lower
This is true for the back-end. As for the front-end, although MVC and MVVM based frameworks are doing a great job providing easier tools writing unit testing, we still see low adoption. Some claim it stems from frontend comprising mostly glue-code, and some say it’s harder to write test DOM and event (asynchronous) code.
The need for front-end automation rises
Nowadays, a huge part of any project resides in the frontend. Not only did past challenges remain, the plot thickens as we have more environments (e.g. mobile and tablet), more Operating Systems (Window & Mac are both popular), and more browsers to support (IE11, Chrome, Safari, Firefox, Opera, Edge, not to mention old versions of IE.. may it rest in peace already).
Percentage of core logic – backend vs. frontend
Since more R&D teams shifted to agile, QA has become the bottleneck. A big part of being agile is the frequent releases. When a six month project comprises a three week testing cycle it’s one thing, but when a team wants to deploy weekly or even daily without bugs, QA is challenged with shrinking that to a cycle that takes 1-2 hours. The feedback to the developer should be instantaneous, as research show that longer the feedback cycle is, the longer it takes to fix the problem.
Why testing frontend is difficult
With the move to frontend, high maintenance overhead resurfaced. The maintenance for frontend test automation was as high as 50% twenty years ago. It has only improved marginally in recent years and is still as high as 30%.
To mimic a user interaction, you must first locate the element you wish to act upon. There are two major approaches to locate an element:
- Visual based – look at the pixels
Frameworks such as Sikuli use that technique, but it’s super fragile, as render per OS/browser/display-adapter will generate different pixels. Some display adapters are non-deterministic in nature. To have stable tests you must be a computer vision guru. - Object based – look at the object hierarchy (DOM).
Frameworks such as Selenium spread like fire. As opposed to twenty years ago, better software design pattern emerged (such as the Page Object) which helped separate the tests’ business logic from the implementation, and urged reuse.
Nonetheless, both approaches result in flaky and unstable tests. They require high maintenance. or every UI (code) change, a test change is required as well (as both rely on the UI). It’s easier for those practicing the shift left paradigm, where quality is inherent in the development cycle, as finding the cause is easier, but they suffer from the same amount of failures. For those with a QA team, it’s still catastrophic. As almost every morning starts with understanding why tests fail and whether it’s a bug or merely a code change.
Crowdsourcing
For End-to-End testing, some companies turn to crowdsourcing to alleviate the pain. This works great for regression tests which require no prior knowledge of the app. The required ramp up is just to write down the instructions in plain English. The big advantage is the low cost of people willing to click all day long. The problem arises when one tries to implement the shift left paradigm. This requires testing not only in the main branch, but also in all dozens of feature branches. Not only does this makes it not cost effective, but developers are custom to get feedback in seconds or minutes, not hours.
Image a developer promoting a hotfix to be released to fix a critical bug yet having to wait for someone in the other side of the world to wake up and test the new version.
Communication is also challenging at the developer and tester have no experience working together.
As for exploratory testing, crowdsourcing still great. It allows one to test different devices and different networks from all around the world. The rose of the garden is the seductive business model, paying only for proven bugs, comprising a thorn where every little bug is reported and the devs find themselves trying to prove that some of them are not real bugs. Until today, crowdsourcing does not provide a sustainable solution when shifting to continuous delivery.
