What are Vanity Metrics?
A vanity metric is a measure of success that we use to delude ourselves and impress upon others the extent of our achievements. The important distinction to make regarding vanity metrics is that they are an introverted measure of success that fails to measure real value from the customer perspective.
To describe velocity in this way may appear to be harsh criticism of an agile practice that is held in high regard by many capable agile practitioners. But in this post I hope to dispel some common myths regarding velocity, and share some experiences regarding better ways by which agile teams can validate the success of their actions.
It is my personal belief that many teams rely too much upon velocity as a measure of success. I base this opinion on the premise that velocity is an introverted measure that is a calculation of progress against remaining estimated effort or complexity, and it is the following two principles that form the basis of my argument:
1. Velocity is a calculation based upon estimation
As humans we are predisposed to be incompetent when it comes to estimation of risk, effort and uncertainty. We are riddled with cognitive bias that affects our ability to provide a logical and reasoned prediction. We are each hardwired to impose irrational and emotional influence when estimating our capability to achieve objectives and the amount of time and effort involved.
Jim Benson’s book Why Plans Fail provides some interesting insights into cognitive bias, and so does the work of Linda Rising. It was at a recent event where I heard Linda speaking that she delivered a quote that resonated with my own views on estimation:
‘An estimate is a guess it is not a number; you can’t add, subtract, multiply or divide a guess.’
When recently interviewing Jim Benson for the IT Kanban Podcast he also provided comment I found equally insightful:
‘We took something as inherently variable as an estimate and we gave it an integer value.’
When we consider velocity from this perspective it starts to appear less reliable as a means to demonstrate success. It is a calculation based upon guesswork, and one that provides an integer value to communicate something that is inherently variable. It is also a metric that is subject to influence and bias from within the development process.
Some teams even pursue the goal of hyper-productivity, where velocity reaches previously unobtainable levels of burn-down, that is massive amounts of progress against guesswork described as an integer value. This is what we mean by vanity metrics; this metric is of no perceivable value to the customer or consumer.
2. A reduction in remaining effort is of no direct value
Reducing the amount of work remaining on a product backlog or list of requirements does not directly demonstrate value. It may or may not be demonstrative of progress towards something that will be of value to our customer, but how do we know?
The end users and consumers of the products we develop have no interest in the amount of remaining work on our product backlog. Customers only care about the features we have already delivered into production. What difference does it make to our customer whether we have delivered 5% or 90% of our potential list of requirements?
Customers value the features we have already delivered. As the producers of IT products and services we typically value the affect those features have had on customer behaviour, such as increased customer retention and increased levels of interaction with our product. A calculation of guesswork regarding the amount of work remaining, expressed as an integer value, is not a measure of customer reaction, and we can find better alternatives.
My suggestion is that you consider alternatives to velocity, and measure your progress against a clear set of actionable metrics that are a measure of customer interaction with your product. Try this with your development teams and stakeholders and discuss what you find. Actionable metrics can provide real insight to the way customers value your product, and can significantly improve prioritisation.
I’ve seen actionable metrics focus the attention of development teams on valued customer behaviour. The ability to influence customer behaviour through the development of product enhancements is far more beneficial than the ability to estimate with predictability or burn through tasks at hyper-velocity.
I would summarise my views like this: actionable metrics measure progress, and velocity measures activity. I’m interested in progress, and so are your customers.