I had a few bad experiences with metrics early in my career that made me leery of metrics in general – so leery that I have to confess to shying away from almost all of them. I was fine with the standard agile metrics. Burn-down charts and velocity seemed harmless enough – but beyond those? The thought of proactively using metrics and reports to illuminate issues and drive improvements would always trigger worries about how drastically things could go wrong.
Ineffective IT help desk metrics
When I first joined one company, I couldn’t get into my voicemail. The light on the phone was blinking, but I couldn’t access the message or messages that were waiting for me. I followed the instructions but with no success. So I opened a ticket. The ticket went off to a centralized support organization thousands of miles away. The support agent emailed more instructions and closed the ticket. Unfortunately, the new instructions didn’t resolve the problem, and there was no way to re-open the ticket, so I opened a new ticket.
We went through the same dance again. The agent said that a change had been made and that it should work now and closed my second ticket. Again, no dice. I opened a third ticket. I was starting to get quite frustrated now. It took a fourth ticket before the agent finally fixed whatever it was that needed fixing. While my problem was eventually solved, my satisfaction in the support I received was hovering right around zero.
What happened here? It’s only a theory and I have no real proof, but my guess is that this support organization was primarily measuring its success – and the success of its people – by how many tickets they were closing and how quickly they could close them. They were focusing on metrics like Average Handling Time (AHT) or Mean Time to Resolution (MTTR) without taking into account whether or not the customer’s problem was really being solved.
Closing tickets quickly but unsatisfactorily was actually being encouraged in this system – or certainly not discouraged. The fact that it required four tickets to resolve my issue meant that their stats showed four resolved issues – even though only one actual issue had been resolved. The measured AHT or MTTR would have been short, even though it really took, if I recall, about a week to actually resolve the original issue.
It’s not that AHT or MTTR are bad metrics. The problem is, used incorrectly and in isolation they can drive bad behaviour instead of good behaviour. Without other metrics to provide a more holistic view of how this support organization was performing, it was too easy to encourage the wrong behaviours instead of the right ones.
Counter-productive quality metrics
In another organization, the VP of Engineering was keen to improve quality; certainly a worthy goal. His metrics measured the number of Critical and Severe bugs that were found internally and compared those numbers to the numbers found by customers. The QA group was praised for finding Critical and Severe issues before they made it to the field. The Development group was chastised for creating critical and severe issues. The QA group was chastised for critical and severe issues that made it to the field. The result? There was a lot of non-productive arguing between QA and Dev on whether an issue was Critical or Severe as opposed to Normal or Minor. QA kept trying to elevate the severity of issues and Dev kept trying to downplay their importance. Then there was more un-productive arguing between the support organization and QA about the severity of customer issues.
Again, the motivations behind the metrics were good, but the resulting behaviours were counter-productive and didn’t noticeably help us to improve quality. My feeling at the time was that we would have been better off with no quality metrics at all. Cycles wasted on arguing would have been better spent elsewhere.
A bad system
A bad system will beat a good person every time
— W. Edwards Deming
I can think of many other similar examples during my career where metrics encouraged behaviours that were ineffective and counter-productive. The stronger the focus on the metric the more risk that it would distort behaviour for the worse instead of aligning behaviours for the better. This seems to especially be the case when bonuses or other financial incentives are involved.
It might be tempting to blame the people in these cases. How could an IT support agent think that it’s the right thing to force people to keep opening new tickets for the same issue? Why can’t people just agree on the definition of a Critical or Severe issue and move on? If it was only one or two individuals behaving badly I might agree that those people need some coaching. What I’ve seen more often though is evidence of a systemic issue – in other words, the problem is the system and not the people.
A force for good
What’s measured improves
— Peter F. Drucker
As I confessed at the start of this article, all of my negative experiences led me to shy away from using metrics as a tool to help drive improvements. I now know that was a mistake. While metrics done poorly can cause bad behaviours, done well they can be a powerful tool to for good. Measurements and metrics can be an invaluable tool to inform and guide better decisions.
It is possible to continuously improve without a lot of support from metrics. Retrospectives can bring out pain points, and action plans can be formed to improve things. The next retrospective can review whether the changes seemed to improve things or not based on the subjective experience of team members. This at least provides some form of feedback, though subjective, on whether or not things are moving in the right direction.
Metrics can give a more objective and quantified view of your reality. They can take abstract concepts and make them more concrete. When you say “productivity improved”, by how much? If you say “a lot”, does your version of “a lot” match my version of a lot? Translating feedback into numbers clarifies that feedback and can make it more powerful and compelling.
When you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind; it may be the beginning of knowledge, but you have scarcely in your thoughts advanced to the state of science.
—Lord Kelvin (1824–1907), British physicist and member of the House of Lords
Metrics without a purpose are just noise
Measurements must serve a purpose to be of value. There’s no point in putting together a lot of fancy tables, charts and graphs if you won’t be making decisions or taking actions based on what you see. Measurements that matter answer questions that matter.
If a measurement matters at all, it is because it must have some conceivable effect on decisions and behaviour. If we can’t identify a decision that could be affected by a proposed measurement and how it could change those decisions, then the measurement simply has no value”
― Douglas W. Hubbard, How to Measure Anything: Finding the Value of “Intangibles” in Business
Many people recommend that if measuring something comes for free, then you should do it. You may not need the measurement now, but you might later, so you should collect the information even if you don’t currently see the value. I will clarify their advice by emphasizing that they are talking about data collection, not data analysis and interpretation.
Collecting raw data that you can mine later for information makes sense if that data can be collected for free. Reporting on that data – especially if those reports will distract from other more important reports – doesn’t make sense if you aren’t trying to answer a question or solve a problem. Don’t clutter your primary dashboards with useless reports. Focus on what matters.
Improving our metrics
So how do we avoid counter-productive metrics? If metrics are so powerful, how do we ensure that they work for us and not against us? If you are responsible for measurements and metrics, how do you detect when things are going off the rails and course-correct?
First, before you roll out any new metric, consider how people might be tempted to “work to the metric”. What kinds of unintended consequences could the metric have? If you can imagine some, are there other complementary metrics that would reveal if these things were happening? If so, build your metrics to include the necessary checks and balances.
For example, in the IT support case described above, a focus on First Call/Contact Resolution (the percentage of times that issues are resolved with a single call/contact) might have provided the necessary checks and balances against people trying to improve Average Handling Time without a good system level view. An emphasis on customer satisfaction metrics would also drive better behaviours.
As for Average Handling Time and Mean Time to Resolution, you need to ask yourself what your purpose is in reporting on those metrics. If AHT or MTTR are increasing or decreasing, how should that be interpreted? For example, an increase in Average Handling Time might be a good thing if it means that the quick-to-solve problems are being eliminated by improving the usability of the product before it’s released. It might also indicate that the users themselves are able to solve easy problems by searching forums and knowledge base articles without calling or contacting support. On the other hand, it might mean that the product is becoming harder to support, or that the support agents need additional training and information on how to troubleshoot issues. Without more information, it’s impossible to interpret if movements one way or another are good or bad.
If you’re collecting and reporting on AHT or MTTR as a measure of the effectiveness of your support organization, then hopefully you can see that this measure can be counter-productive without compensating metrics that look at the system as a whole. If you still aren’t convinced, check out this article on the subject.
On the other hand, AHT and MTTR can be useful measures if you’re using them to help forecast required staffing levels or to identify trends. This brings us to my second piece of advice. Let’s imagine that you’ve measured something for a while and you can see a trend over time. Maybe the trend seems to align with your goal and maybe it doesn’t. In either case it’s time to do some more analysis. Does the trend mean what you think it means? If you’re uncertain, what additional measurements can you collect to become more certain?
My third piece of advice is, if people are complaining about a metric that you’re collecting, be sure to listen and dig further. As much as you may have tried to construct your metrics carefully, if you’re hearing complaints it’s likely you missed something. Be open to feedback and learn from it.
The power of metrics done right
Used properly, metrics are a powerful tool in the quest for continuous improvement. People and teams crave feedback. They want to know that their work matters. Carefully constructed metrics can help people see that their efforts are making a difference. Metrics can shine a spotlight on great team performance and point out areas where more effort can make an even greater difference. They help us quantify our successes and learn from our failures.
The fact that poorly conceived metrics can cause problems is just more evidence of their power to drive changes in behaviour. The problems that can occur due to poorly conceived metrics can be avoided or resolved by anyone willing to put in the necessary thought and effort. Avoiding metrics out of fear is not the right answer. We have a responsibility to our teams and our organizations to find ways to use the power of metrics for good.