CI/CD transition: Common mistakes to avoid & tips for a successful transition

CI/CD, BDD, ATDD, DevOps, agile, continuous testing—however you want to do it, companies today are all striving for faster release cycles and higher quality software. While helping our clients over the years, the teams behind Testim.io and Abstracta have witnessed several successful transitions to continuous integration and continuous delivery (CI/CD). It’s important to remember that change requires not only a technical shift, but also a cultural one at the same time in order to create a lasting impact. We’ve identified eight common mistakes that teams make during the transition that you can avoid while embarking on your own journey transitioning team culture, processes, expectation setting, and more.

Mistake 1: Lacking Proper Management Buy-In

A Bottom-Up Endeavor

More often than not, the driving force behind the shift to CI/CD is a bottom-up effort, meaning that leaders seldom impress upon their teams the need to adopt these new practices. Rather, testers and developers make the case for it. The problem with implementing CI/CD from the bottom up is that there comes a point when the team must obtain executive buy-in to move forward. Otherwise, the project may gain some temporary momentum, but it’s unlikely to be sustainable over time. Although partially making the shift is better than nothing at all, it’s ideal to provide management with visibility into the team’s success in order to receive their support and to spread CI/CD to other teams throughout the organization.

When it comes to a bottom-up implementation, the most vital factor for a team’s successful shift to CI/CD is its ability to showcase the value of the adoption to management.

Increase the likelihood of management buy-in

Especially when working inside a fairly rigid organization, it’s very rare to see the shift to CI/CD occurring top-down. In these cases, one effective strategy is for the technical team to align itself and sync up in terms of implementing continuous integration, adopting CI’s practices and a DevOps culture, and agreeing on some of the key benefits that these practices will present to management as a result of the changes. For example, one potential benefit may be finding issues earlier in a process, thus being able to identify and eliminate risks much sooner and at a lower cost.

It’s important to do frequent reviews that allow the team to have better visibility of these issues in an earlier stage. It’s also practical to properly implement a development strategy that allows the team to have a clear process and trackable metrics for improving it over time.

The shift to CI/CD is usually triggered from the bottom up, but at the end of the day, it’s a transformation that requires management buy-in because it’s about skill set, processes, and several areas to which management needs to allocate the proper time, resources, and attention.

Mistake 2: Underestimating the Importance of the Necessary Cultural and Mindset Shifts

The Responsibility for Software Quality

In the original illustration by Dan Ashby above, one can see that there is no specific time for testing in DevOps. Meaning, testing should be a fundamental part of every single task, making it a continuous activity. In CI/CD, it’s a part of everything that development teams do and may be part of the mindset shift that whatever phase of development one is currently in, there must be some sort of testing included as an integral part of it.

Testing should be something that everyone on the team is doing all the time, as a part of all their activities.

When it comes to software quality, all team members share ownership of it. For instance, the developers can write unit tests and also write the code with testability in mind, helping to mitigate risk from the start.

One simple way to help reflect the change in the view of testing is to change the testers’ titles from QA to software tester or better yet, quality engineer. While this change may seem surface-level, too simple, or even silly, if someone has the title of “software quality assurance,” it conveys the wrong message about who has the responsibility to ensure the product quality (which in agile, CI/CD and DevOps is everyone).

Another fundamental aspect is to be aligned in terms of quality, what quality means for the organization, the stakeholders, the team as a whole, and each team member.

Continuous Feedback Culture

Culture is key, and it needs to be conducive to a high level of collaboration and communication. In DevOps, the most important element is the constant giving and receiving of feedback. In our experience, you cannot claim to have a culture of DevOps or have CI/CD in place if you aren’t holding retrospective meetings.

Most likely if a team is trying to adopt CI/CD, then it is adopting an agile approach because it is not something that teams consider when following a more traditional approach such as waterfall. In agile, and particularly in scrum, retrospective meetings are fundamental for getting teams accustomed to giving and understanding feedback, allowing for continuous improvement.

The first step for adopting an agile methodology is to get into the habit of holding retrospectives, aka, “retros.” In the first retro, one of the action items is to set up recurring retro meetings every two weeks. In that way, the practice of exchanging feedback becomes continuous.

The Definition of Done

Here, the question of the matter is, When an item is in the “done” column in the team’s Trello or Kanban board, what does everyone understand was executed for that item?

Does everyone have a common understanding of the team’s “Definition of Done” (DoD)? Can it be assumed that the item was tested and how it was tested? Can it be assumed that someone reviewed the code? What else?

The DoD is a very powerful tool to share this understanding of the quality standards of what the team produces and how. It provides visibility into the process, even including how to implement CI/CD, as well as fosters trust. This is part of the mindset shift that must take place. If the DoD is unclear, the team must define it together and make it visible and accessible to everyone.

For a successful shift to CI/CD to occur, it’s important for the team to adopt an agile mindset and culture, starting with a shared responsibility for software quality, fomenting a culture of continuous feedback firstly by conducting retro meetings, as well as having a transparent and universally agreed upon “Definition of Done.”

Mistake 3: Not Setting Clear, Defined, and Realistic Goals

What to Aim For?

For the success of any type of transformation to occur, it should be measured and comprise of tangible goals. When defining goals, it’s very important to think about where the organization stands. If the team is currently releasing once a month, a realistic goal may be to cut down that time to just two weeks. A different, yet respectable goal for the same scenario would be to continue to release once a month but to do so with greater quality and less risk.

While it may not quite be possible to imitate a company like Amazon and release every 12 seconds, teams can consider where they are today, and at the very least, think about assessing their operating reality.

In thinking of their goals, teams need not to only think about what’s realistic (what is truly achievable?) but also what will make the most sense at the present time. All of the goals that are decided upon should be adjusted to the team’s current context – for example, considering how frequently users need new versions, adjustments, or fixes.

For some teams, it may not make sense to release more than three times in a year, and that’s fine. In this case, it would be a mistake to over-engineer processes if they do not produce any added value.

Always keep in mind what is going to make the most sense. Will continuous delivery or just continuous integration alone be adequate? In the case of highly regulated organizations like banks or healthcare industry companies, CI alone may be sufficient, without the need for reaching continuous delivery.

In our opinion, continuous integration should be a must for every team, as it tends to become the gold standard for any team that experiences it.

Unlike CI, CD fundamentally changes the way companies deliver software as it impacts users, the business, and operations. Normally, teams should aim to implement CI as a starting point. Once implemented, it is possible to think about how to start releasing faster, and if doing so is ideal. Try it for a couple of sprints, measure it and evaluate how well it’s working in order to make a long-term plan.

Determine whether your team will benefit from achieving continuous delivery or if continuous integration on its own is sufficient for your needs.

Mistake 4: Failing to Make a Scheduled Plan with Deadlines

The Blueprint

It can be really difficult to make a plan for something that implies a fair amount of research and the adoption of various tools. Before you make a plan, understand what the CI process entails, as illustrated in the following graphic:

In CI, there is a separate server that fetches the new changes from the source control server which makes a new build and tests it for each change in the code. This is done frequently — at least once a day, if not more.

When teams prepare the build automatically, they run the different tests that they’ve prepared and everyone receives a notification if anything has gone wrong or if everything is okay with the new build. This is part of the continuous feedback element.

This is the blueprint for the first step to undertake when making the transition

Create MVPs

In planning the transition, your team may ask, “What is a minimum viable product (MVP) for this change? What makes sense and adds value to the delivery pipeline?”

An initial goal could be continuous integration, starting with the automation of the build and some unique tests, and after that, the team can evaluate and plan the next MVP, adding some other forms of automation to the process. Once CI is in place, the team can aim for continuous delivery.

It is important to note that the process itself must be tested in order for it to be trusted. Testers need to trust the delivery process and the automation that is involved there since therein lies great risk. The second step may be to define a plan for testing the process.

There are countless aspects to consider when defining a pipeline. In this stage, testers can add a lot of value and contribute ideas about which verifications to add to the testing process. Also make sure to ask questions like “Who is going to prepare each test?” and, “In which environment will we run those tests?”

A good process to have in place is one in which each developer runs automated tests at different levels, (like at the API or UI level) in their own branch and after that, they ask for a merge request and someone else then performs a code review. After that, all the tests could be run in a fresh environment generated for the main branch. There the team can also include performance verifications or security checks or use tools to find issues related to accessibility, among other things.

Follow an agile approach for planning the transition, creating a series of “MVPs” organized in different sprints, adding different levels of automation to the delivery pipeline in each.

When it comes to timing, it greatly depends on the technologies or the tools to be used, as well as the skills of those on the team.

Plan for Managing Costs

Another important consideration when you’re building the CI process is cost. To manage this, some teams segment their tests into buckets, essentially running different buckets at different stages in the development process. It’s possible to run an entire regression suite on every commit or every merge but, unless the team is counting on a highly robust environment, it will take some time to get feedback to the developers. Many teams set out to have a small subset inside a branch and then, once they merge the master, have a sanity suite that is robust enough but still allows the developer to get feedback in a few minutes and then once a night, run a full regression. Testim.io follows this practice to balance the cost of infrastructure with agility.

Know what the CI/CD process will entail and create a plan for the transition involving a series of MVPs for each stage while keeping in mind how to manage costs. The timing will depend on each team and the tools they choose to use.

Mistake 5: Confusion About Role Changes and Responsibility Shifts

Each transition plan includes different people with different roles. Teams must manage the plethora of changes to be made and shifting responsibilities. First, figure out what processes, people, tools and technology are needed to reach your goals. Determine what will stay the course, what will be modified or eliminated, and what should be added.

Not everything needs a complete overhaul.

For teams that are used to working in agile, there will be fewer changes to the roles and responsibilities. Teams not yet accustomed to being agile will first have to adapt to the changes that come with it. To reiterate, the main mindset that needs to change to be agile is that quality is the responsibility of the whole team. Otherwise, you will have a test team that will work as a quality gate with an inherent conflict of interest with the development team (developers being the builders while testers being the “destroyers” of their work). This conflict of interest between developers and testers was something that was made on purpose for traditional development environments to function, yet it’s important to break out of this mindset in order to transition to CI/CD.

A testament to the shift-left movement gaining traction and developers also being responsible for quality is the fact that 30% of Testim.io users are actually developers, not testers

Essential Technology

Regarding technology, it’s essential to use a git-based repository. It’s not necessary to automate the process from the changes in the repo to the deploy, including all the automatic checks in the middle, the building process, and so on.

Everyone on the team will be responsible for understanding the process.

Continuous delivery is also about not only getting feedback from tools with automatic checks but also from users. Someone must be responsible for checking the logs to see if, for example, the newly released version created any negative impact on the system’s performance or functionality. It’s important to be familiar with application performance monitoring (APM) tools to see if there is any change in the way the resources of the service are being used, if there is a new exception appearing in the log, or if there is a new JavaScript error. There are several well-designed tools on the market today to help process this information.

Mistake 6: Misunderstanding the Technical Requirements

Assessing Your Team’s Maturity

How do you determine the technical requirements and the tools needed to make the shift to CI/CD?

An important first step is understanding where your team is today. Based on that, it’s possible to plan what needs to change and what or who needs to be involved. The plan should also be linked, of course, to the team’s goals and expectations. From experiences working with several clients, Abstracta has identified three pillars that teams must assess if they want to make improvements: people, technology, and processes (and how they interact together).

In doing this, software development teams assess their maturity, and what they will place the most emphasis on, their testing team and testing maturity, which is the main driver of CI/CD and DevOps.

Abstracta has established a testing maturity model, wherein there are three main levels of testing maturity based on three basic characteristics; risks, quality, and costs. The model contains everything deemed necessary in order to lay the foundations for efficient testing, which is the minimum level of testing maturity that makes it possible to achieve CI/CD.

Using this chart is a great way to trace all the activities to carry out and to define some preconditions for making the transition. It allows for a broad overview of a team’s technology landscape and it helps to determine which activities to engage in.

You can identify your testing maturity level using the chart below or take this online software testing maturity assessment.

 

The 3 Levels of Testing Maturity

1. Basic Testing

The first level of maturity in this model is “basic testing.” At this level, there are different activities that need to happen in terms of the source code. For example, it’s necessary to have versioning as well as a separate environment from which the code can be tested. Bug tracking should be in place as well as test management. Teams at this level know what it is they are testing and have achieved traceability between test cases and product features. In terms of automation, they may have at least some unit testing in place and test at the API level. This level of maturity may also comprise some performance and security testing.

2. Efficient Testing

One level ahead of basic testing, “efficient testing” is characterized by the achievement of a more efficient way of testing where various aspects like risk are not only identified but controlled. At this level, teams begin to have more coverage in terms of testing, with more data management, and they may even have separate environments added. Some optional areas of quality here include defect causal analysis, greater prioritization in testing, and regarding automation, more end-to-end testing.

3. Continuous Testing

When a team reaches continuous testing, continuous integration is already set in motion — and maybe even continuous deployment, depending on the team’s needs. Teams at this level have a set of unit, API, and automated tests that are running continuously, generating the highly sought-after continuous feedback loop. Performance tests are also continuous, and hopefully, there are also security and accessibility checks included in the pipeline. At this level, risks are reduced and quality and costs are optimized.

Using this testing maturity scheme, it’s possible to understand what are the necessary technical requirements and preconditions in order to advance to CI/CD, as continuous testing is a primary enabler of CI/CD.

Mistake 7: Forgoing an Effective Test Automation Strategy

One of the mistakes that teams often make regarding test automation is wanting to automate everything and have every test running all the time. This might not be necessary, so it’s important to define a clear strategy for how to approach automation.

The Test Automation Pyramid

One guideline to follow is the Test Automation Pyramid by Michael Cohen, which illustrates the number of automated tests to aim for across different test levels:

Unit Tests

At the base of the pyramid are unit tests. The aim is to devote more effort to this kind of test because they are faster to execute, easier to maintain and manage, and enable teams to detect errors when they are still premature, leading to a higher ROI. Developers should be in charge of writing these whenever they’re writing a new feature, cutting short the lifespan of any bugs that may appear.

Acceptance, Integration, and Component Tests

Moving to the next level are acceptance, integration, and component tests. Considerable efforts should be directed here, which is automation at the service level. When it comes to API testing, only the most critical workflows should be automated, in order to test the logic before reaching the UI. At this level, the tests are slower than unit tests but are still quicker than UI tests, so they will help to provide good feedback on how different services are working.

Automated GUI Tests

The top of the pyramid represents having a few automated tests at the UI level. These are the slowest and most difficult tests to maintain, but they serve the purpose of validating end-to-end functionality. These tests replicate how a user would actually use the system. When a team reaches this point and establishes a good foundation for its automation strategy, there will still be a need for manual and exploratory testing.

Cohn’s pyramid is mostly focused on functional testing, but it also recommends including unit performance checks to the pipeline.

But, don’t forget—it’s also important to dedicate time and effort to maintain the automated tests so that their results remain reliable.

Testim.io has helped several developers and testers instill the habit of running automated functional and end-to-end tests because it uses artificial intelligence to keep the tests maintainable.

Take a Risk-Based Approach

Instead of boiling the ocean with your automation strategy, take a risk-based approach, focusing on the areas that will result in the highest return on investment (ROI).

For example, for an e-commerce business, a critical feature to test is the online check-out process, as that user experience will have a direct impact on the business and its bottom line. For a search engine, the search functionality would be the main focus of testing

Focus on the user scenarios that are critical to your business and those that are critical to the user experience.

Larger organizations are putting more focus (once they have a good baseline) on the visual aspect of their applications. Today, much more attention is placed on the user experience and the look and feel of applications. There’s a multitude of tools to add different tests or checks to the pipeline, whether they be visual, performance, security, or accessibility checks. For all of these checks, it’s important to take a risk-based approach because teams can get carried away and there is a considerable amount of preparation and maintenance to be wary of. Pay attention to what is truly adding value to the pipeline and helping to avoid risks.

Test automation is a crucial aspect of CI/CD. To know what to automate and where it is a good practice to follow the approach illustrated by Michael Cohn’s Automation Pyramid and prioritize the tests to be automated using a risk-based approach.

Mistake 8: Failing to Set the Right KPIs to Measure Success

How do you know if your transition is going well? How do you know if your continuous integration or continuous delivery environments are producing the expected results?

What we have seen at Testim.io and Abstracta, particularly in the CI/CD context, is that teams should pay attention to different metrics associated with the delivery pipeline. One of the most important questions related to CI/CD is, “How much time does it take for a change to reach the user?” Teams can use that information to start thinking about how they can reduce this time.

There are some useful metrics related to the lean methodology: lead time and cycle times. These are related to how much time is needed to go from an idea to implementing and delivering that idea to the users.

Other metrics that are always paramount, regardless of whether a team is working in CI/CD or not, are how well the users rate the software and how happy the team is with itself and the way it works. Those are two fundamental key performance indicators (KPIs) as the basis of any company is a group of people whose mission is to deliver value to customers.

If a team wants to utilize a dashboard, it can turn to its CI engine (like Jenkins) which may provide more information related to the technical aspects of the CI/CD pipeline, like the tests that are passing or failing, response times, code coverage, etc.

The important thing that teams and leaders should always be mindful of is how much time it takes to deliver value. This is the measure of success for implementing CI/CD.

About Abstracta

Abstracta is a world leader in software testing and quality engineering focused on improving the performance of software applications, implementing agile testing, and setting up automation stacks in CI/CD environments.

With offices in Latin America and Silicon Valley, Abstracta has expertise working not only with leading-edge proprietary and open source testing tools, but developing specialized tools for the financial, retail and technology sectors including companies such as BBVA Financial Group, CA Technologies, and Shutterfly.

Abstracta was recently named the fifth best software testing company for 2019 by Hacker Noon and is a top-rated outsourcing company according to Clutch.co.

About Testim

Software is changing the world every day. It’s behind everything we do from powering back office systems and websites to robotics and mobile apps used across a wide range of devices. We want to help people developing powerful applications that are improving everyone’s lives.

We created Testim because automated testing is still difficult. Having been developers for the last 20 years, we recognized that we spend time and energy maintaining our automated testing environments, but are still anxious with how a simple bug fix might break another part of our application.

Testim is software that’s easy and works the way you want it to, supporting agile transformations and shift left.

Our AI-based software testing platform uses machine learning for the authoring, execution, and maintenance of automated test suites to support your organization’s user experience for every release.