Back to blog

Agile Transformation: 3 Critical KPIs to Measure Success

It’s inevitable. Agile has matured and in addition to speed, now the new set of challenges are about accountability and…

By Testim, August 16, 2017

It’s inevitable. Agile has matured and in addition to speed, now the new set of challenges are about accountability and proving its worth. Just as with other areas of the business, Agile must also answer to traceability, auditability, and compliance. After all, what good is a methodology that delivers fast, but ultimately fails to deliver more value? Many organizations are now demanding it and teams new to Agile are starting out in an uphill battle with risk-averse management and globally dispersed project teams, in addition to technical debt. Even though the executive team made the decision to adopt Agile practices, that decision is coming with its own set of expectations that must be met. And it’s qualitative metrics that will be looked to to satisfy this expectation.

So which metrics should you put in place? Is it more effective to track activity within Agile tools, such as JIRA? Or is is better to track metrics within the software itself? The important thing to realize, though, is what’s really being asked: “is Technology actually improving its impact on the business in a tangible way. Or said another way, as phrased by the Kanban community, is Technology IT becoming more fit for purpose? Answering this question, of course, requires a clear understanding of what is that the Business expects from its interactions with IT.

To ensure metrics remain relevant, they should be inspected and potentially evolve to ensure they still align with the organization’s short and long term goals. Certain metrics may be created to align with the organization’s specific agile maturity phase and as an organization moves from one phase to another, certain metrics may no longer be relevant. Inspection and adaption will ensure metrics align with the specific goals.

Measure the amount of working software value that reaches customers hands, such as “percentage of story points completed within a sprint”. Velocity trends (not absolute values) are also helpful. Although using Story Points is a crude way of doing this, much better to also estimate the business value of each story (so each story gets two estimates, one for complexity and one for value) then you can measure the value produced by the team easily. The tough part about value is – it is not an easy thing to assign the value to a story. In some circumstances, value cannot be a KPI for IT projects because (1) business value is subjective especially if business value points are used; (2) some projects have no direct way to calculate business value (think regulatory projects that must be done to comply with regulations); and (3) Business value varies too much from project to project based on the perceived business benefit making it hard to measure objectively.

Agile Transformation KPIs

That said, KPIs to track during a transformation to Agile include those that fall under the categories of time, quality, and effort. Instead of measuring traditional key performance indicators (KPIs) like profitability or transactions per hour, therefore, executives must shift their focus to how well the enterprise is able to deal with changing customer preferences and a shifting competitive landscape.

Quality

In terms of quality, nothing measures a product’s value or worth better than ROI. The trick is to make sure the return is defined clearly and is measurable. The most straight-forward way to do that is to tie it to a number value: sales, downloads, new members, positive comments, social media likes, etc.

The number of defects is a simple metric to track that speaks loads about not only the quality of the product, but also team dynamics and effectiveness. Good metrics to track for defects include, the number of defects found during development, after release, or by customers or people outside of the team. It can also be insightful to track the number of defects that are carried over to a future release, the number of support tickets, and the percentage of manual vs automated test coverage.

Effort

Sprints are fundamental to Agile iterations. KPIs, therefore, become fundamental to sprints. The design phase of sprints is initial to the development process and, while tweaked as needed to remain focused on objectives, is to remain in place unless proved to be unreliable in achieving objectives.

Velocity tracks the number of user story points completed per sprint. It’s effectiveness to depict anything of value outside of the technology team is debatable. It’s often a misunderstood metric because it’s confused with a team’s productivity. It’s not uncommon for executive management to demand a higher number of story points be completed per sprint in an effort to increase speed to market. However, more often than not this results in a lower quality product. Nonetheless, it can be useful if it’s understood and reported on as an effort-based metric instead of time or quality.

When measuring Velocity, it’s important to keep in mind that different project teams will have different velocities. Each team’s velocity is unique. If team A has a velocity of 50 and team B has a velocity of 75, it doesn’t mean that team B has higher throughput. Since each team’s estimation culture is unique, their velocity will be as well. Resist the temptation to compare velocity across teams. Measure the level of effort and output of work based on each team’s unique interpretation of story points.

Time

Scrum teams organize development into time-boxed sprints. At the outset of the sprint, the team forecasts how much work they can complete during a sprint. A sprint burndown report then tracks the completion of work throughout the sprint. The x-axis represents time, and the y-axis refers to the amount of work left to complete, measured in either story points or hours. The goal is to have all the forecast work completed by the end of the sprint.

A team that consistently meets its forecast is a compelling advertisement for agile in their organization. But don’t let that tempt you to fudge the numbers by declaring an item complete before it really is. It may look good in the short term, but in the long run only hampers learning and improvement.

SUMMARY

KPIs are vital to the strategic goals of an organization as in addition to revealing whether the direction of project is on-course, they assist in informing key business decisions. Metrics are just one part in building a team’s culture. They give quantitative insight into the team’s performance and provide measurable goals for the team. While they’re important, don’t get obsessed. Listening to the team’s feedback during retrospectives is equally important in growing trust across the team, quality in the product, and development speed through the release process.