Business environments are drastically different today than what they were a few decades ago. This is especially true for tech companies. Adapting to an uncertain and even turbulent business environment is the key to success in the tech industry. That’s why companies are constantly looking for methods that help them identify uncertainty and adapt and respond accordingly.
Agile is one such methodology that has been around for more than 15 years now. It focuses on building autonomous, high-performance, and cross-functional teams that are capable of adapting quickly to customer feedback and changes in the business environment. Simply speaking, Agile is an alternative approach to project management that:
The Agile methodology was developed in 2001 as an alternative to traditional waterfall methods. Waterfall project management doesn’t allow for any changes to take place during the development process. No step can begin until the previous step has finished, and no step can be revisited later. As you can imagine, the Waterfall is completely inappropriate in a business environment that thrives on adaptability. Hence, the need for Agile.
Agile metrics: the key to successful Agile implementation
Despite its numerous benefits, implementing Agile in your organization can be tricky, especially if you want it to have a transformative impact. Measuring Agile success is a critical part of effective Agile implementation. Here’s why:
Cost of implementing Agile
The cost of change that comes with adopting Agile can be significant. Of course, this includes the tangible costs of training, reorganization, coaching, and re-engineering of the development environment. It also includes intangible costs like productivity losses that are an inevitable part of such massive change. Since the organization is undertaking such a significant investment when it shifts to Agile, it’s very important to measure how the Agile implementation is going and if it is actually yielding significant returns. This is where effective Agile metrics come into play.
Agile and the PDCA Loop
Agile can only be effective if it follows the PDCA(Plan-Do-Check-Act) Loop that most lean approaches use. For effective Agile implementation, you need to first set a goal and decide what needs to be done to achieve it (Plan), get it done (Do), inspect the results (Check), make the adjustments you need (Act) and then do it all over again. With every iteration, you get better and move closer to your goal. However, unless you have a mechanism in place to inspect results, the PDCA loop won’t close and the continuous improvement which is the essence of Agile will become impossible. That’s another reason why Agile metrics are so critical.
Measuring Agile success: how to start, things to keep in mind
At the end of the day, the goal of Agile is to satisfy customers by rapidly delivering valuable software. This overarching goal should form the basis for all the metrics that are used to measure Agile success. Here are some of the most important things you need to keep in mind before you decide which Agile metrics makes sense for your business.
In recent years, the value that the product brings to the business has become the most important factor in determining Agile metrics. In fact, 44% of respondents in the State of Agile survey said that they measured the success of their Agile initiatives by their business value. It’s important that all Agile initiatives support corporate strategy to the maximum possible extent.
In some cases, business value is easy enough to ascertain. For instance, if you’re building a product that you are required by contract to deliver within a specific timeframe to your client, then the product automatically becomes valuable. However, for most B2C and even B2B products, the product value is measured through KPIs like customer acquisition, retention, growth, incremental revenue and so on. These KPIs can change with a change in the business environment so it’s important to keep re-examining them to make sure you’re still tracking the right metrics. An epic cumulative chart flow based on value can help you see how business value is getting delivered as product features get completed.
Some people believe that revenue increases and customer feedback, although important, cannot be the only measure of Agile success because they can be affected by a number of factors. It’s important to also have metrics around product quality that can help you gauge Agile success with more objectivity. In fact, 48% of respondents said they use product quality metrics to determine the success of their Agile initiatives. These metrics can be observed throughout the product development cycle with automated continuous testing and delivery. The best part about product quality metrics is that they directly drive product improvement. Defect tracking charts and testing trend charts can be used to effectively measure product quality.
One of the major advantages of Agile is that it creates self-organized teams that are autonomous and high-performing. Therefore it’s important to incorporate metrics to measure how Agile teams are performing. 29% of respondents use productivity metrics to measure the success of their Agile implementation. These metrics enable you to measure progress over a period of time and give a bird’s eye view of how teams have transformed after adopting Agile. Velocity, iterations to production, and WIP are examples of effective productivity metrics. Just looking at a burnup count of features over time is also a simple but effective way to measure productivity.
On-time delivery is the main goal behind adopting methodologies like Agile and DevOps. A whopping 58% of respondents said that on-time delivery is the main factor when it comes to measuring Agile success. Time-to-market builds can become the difference between being a market leader vs just another me-too product. Burnup and burndown charts are a great way to identify and track the right delivery metrics. Burndown charts measure the amount of work to be completed vis-a-vis the time left to complete it. Burnup charts, on the other, hand illustrate the percentage of work that has been completed with respect to the total amount of work.
39% of respondents said they used product scope and features to measure Agile success. Product scope refers to outlining a goal that is to be achieved in the next quarter. It then helps to track progress over time and see how close you are to achieving that goal. Burndown charts are a great way to track progress over time.
Apart from these major considerations, there are a number of other factors that help decide which Agile metrics to choose. These include positive feedback and customer satisfaction, visibility and transparency throughout the workflow, predictability of the development cycle and work completion, and process improvement. Of course, different values will have different priorities based on the stage of your business and the product life cycle. You need to choose metrics that focus on the values that matter to your organization.
Who are Agile metrics for?
The driving force behind Agile is the creation of self-organizing teams. Agile can only be implemented through a set of diverse, autonomous, and high-performing teams. Unless teams are able to understand how they are doing, they will be unable to figure out where improvements need to be made. Agile metrics help teams to review and reflect on data so that future development cycles can become faster and more productive.
However, it’s important to dive deeper into the metrics to understand why the data is a certain way. Say, for instance, release frequency has reduced in the last month. This may be because the team was working on a particularly complex project or it may be a result of a deeper productivity issue. The distinction is important because the action to be taken in either case is completely different. It’s not enough for teams to just review metrics, it’s also important to understand why they are moving in a certain direction.
The Agile Metrics that matter
Based on the purpose behind them, metrics can be classified into:
Pace and value metrics
These metrics measure the value delivered and the pace of delivery. Most experts believe pace and value metrics are the key to effective Agile measurement. Here are some of the most important ones.
One of the most commonly used Agile metrics, Velocity determines the regular run-rate for an Agile team. When things are going fine, the goal is to keep the Velocity as stable and predictable as possible. Of course, teams may also be looking to use automation tools and process efficiencies to increase the Velocity. To calculate Velocity you just need to add all the story points that were moved to “Done” in a particular sprint. The Velocity is calculated by comparing completed story points in the current sprint with those completed in the previous sprint.
One of the major goals of lean methodologies like Agile and DevOps is to shorten release cycles. Measuring the frequency with which new features and bug fixes are released helps keep a track of this goal. The automation, high-performing teams and process efficiencies that come with Agile should help improve release frequency considerably over a period of time.
Work in Progress refers to all the stories that have been committed but not completed yet. Too many tasks in this category could mean that the team is constantly shifting focus and leaving tasks incomplete. There should be an agreed-upon limit to WIP that should always be adhered to unless exceptional circumstances dictate otherwise.
Iteration complete to production
An Agile iteration is a short period of time (usually 1-2 weeks) where a team takes one or two of their customers’ most important user stories and complete them to create working software. Every iteration should usually result in code that goes live, but it may not always happen or it may take too long. Many teams track the time between the end of the iteration and when the code is in production.
Kanban teams often use cycle time as a key metric and they plot cycle time on a control chart. Cycle time is the time that it takes to move individual stories from “in progress” to “done”. Ideally, the average cycle time should not be more than half a sprint. There are two things that teams consider when they measure cycle time- consistency and span of time. The shorter the cycle time, the higher the throughput; and the more consistent the cycle time the more predictability in delivery.
These metrics focus on product quality and are regarded as a more objective measure of Agile success. Here are a few important quality metrics.
Defect density is the number of defects founded in software during a particular development period. Defect density can also be measured per thousand lines of code (KLOC). It’s important to track defects during the iteration and defects during production as two different metrics. Defects during iteration help decide whether software can be released whereas defects during production help you make decisions about which bug-fixes need to be released.
Percentage of automated test coverage
Automated testing is one of the key advantages of Agile. It reduces cycle time and enables more frequent releases. Measuring the percentage of the code base that is covered by automated testing helps understand how much time is being saved.
These metrics are suited for teams that believe customer and employee satisfaction are at the heart of successful Agile implementation. Here are three important ones.
Customer satisfaction at the team level
Customer satisfaction can be measured at the team level for every iteration. Since there are several iterations happening simultaneously, a simple 1-5 scale or smiley face indicators should suffice.
Net Promoter Score (NPS)
The NPS can be used at an enterprise level. It is an index that measures customer satisfaction on a scale of -100 to 100. The NPS survey asks a simple question “How likely are you to recommend this product to someone else?” The NPS is a simple and effective measure to indicate the level of customer satisfaction. Of course, the average NPS differs from one industry to another.
Since autonomous and high-performing teams are key to Agile success, it’s important to measure employee satisfaction. A 1-5 scale or smiley indicators for every iteration work well in this case too. Employees can either rate their own satisfaction or the satisfaction of the team as a whole. These ratings are good starting points for discussions around satisfaction and cultivating a healthy work culture.
Effective Agile implementation takes time, patience, and most importantly, a buy-in from the entire team. This buy-in simply cannot be achieved unless every employee is convinced about the effectiveness of Agile. That’s why it’s critical to have the right Agile metrics in place from day one. As employees start seeing the impact of Agile on teams and projects, they become more motivated to implement Agile across the company. Choosing the right metrics then tracking them consistently is a very important part of successful Agile implementation.