Is Velocity The Agile Vanity Metric?

What are Vanity Metrics?

A vanity metric is a measure of success that we use to delude ourselves and impress upon others the extent of our achievements. The important distinction to make regarding vanity metrics is that they are an introverted measure of success that fails to measure real value from the customer perspective.

To describe velocity in this way may appear to be harsh criticism of an agile practice that is held in high regard by many capable agile practitioners. But in this post I hope to dispel some common myths regarding velocity, and share some experiences regarding better ways by which agile teams can validate the success of their actions.

It is my personal belief that many teams rely too much upon velocity as a measure of success. I base this opinion on the premise that velocity is an introverted measure that is a calculation of progress against remaining estimated effort or complexity, and it is the following two principles that form the basis of my argument:

1. Velocity is a calculation based upon estimation

As humans we are predisposed to be incompetent when it comes to estimation of risk, effort and uncertainty. We are riddled with cognitive bias that affects our ability to provide a logical and reasoned prediction. We are each hardwired to impose irrational and emotional influence when estimating our capability to achieve objectives and the amount of time and effort involved.

Jim Benson’s book Why Plans Fail provides some interesting insights into cognitive bias, and so does the work of Linda Rising. It was at a recent event where I heard Linda speaking that she delivered a quote that resonated with my own views on estimation:

‘An estimate is a guess it is not a number; you can’t add, subtract, multiply or divide a guess.’

When recently interviewing Jim Benson for the IT Kanban Podcast he also provided comment I found equally insightful:

‘We took something as inherently variable as an estimate and we gave it an integer value.’

When we consider velocity from this perspective it starts to appear less reliable as a means to demonstrate success. It is a calculation based upon guesswork, and one that provides an integer value to communicate something that is inherently variable. It is also a metric that is subject to influence and bias from within the development process.

Some teams even pursue the goal of hyper-productivity, where velocity reaches previously unobtainable levels of burn-down, that is massive amounts of progress against guesswork described as an integer value. This is what we mean by vanity metrics; this metric is of no perceivable value to the customer or consumer.

2. A reduction in remaining effort is of no direct value

Reducing the amount of work remaining on a product backlog or list of requirements does not directly demonstrate value. It may or may not be demonstrative of progress towards something that will be of value to our customer, but how do we know?

The end users and consumers of the products we develop have no interest in the amount of remaining work on our product backlog. Customers only care about the features we have already delivered into production. What difference does it make to our customer whether we have delivered 5% or 90% of our potential list of requirements?

Customers value the features we have already delivered. As the producers of IT products and services we typically value the affect those features have had on customer behaviour, such as increased customer retention and increased levels of interaction with our product. A calculation of guesswork regarding the amount of work remaining, expressed as an integer value, is not a measure of customer reaction, and we can find better alternatives.

My suggestion is that you consider alternatives to velocity, and measure your progress against a clear set of actionable metrics that are a measure of customer interaction with your product. Try this with your development teams and stakeholders and discuss what you find. Actionable metrics can provide real insight to the way customers value your product, and can significantly improve prioritisation.

I’ve seen actionable metrics focus the attention of development teams on valued customer behaviour. The ability to influence customer behaviour through the development of product enhancements is far more beneficial than the ability to estimate with predictability or burn through tasks at hyper-velocity.

I would summarise my views like this: actionable metrics measure progress, and velocity measures activity. I’m interested in progress, and so are your customers.

More on this topic:

Linda Rising : Agile Connect 2011 – Keynote: Deception and Estimating – How We Fool Ourselves

Jim Benson Interview Personal Kanban and Cognitive Bias

5 Vanity Metrics to Stop Measuring (And Better Alternatives)

Vanity Metrics vs. Actionable Metrics – Guest Post by Eric Ries

(Founder Stories) Eric Ries: On “Vanity Metrics” And “Success Theater”

Don’t Be Fooled By Vanity Metrics

Why Plans Fail: Cognitive Bias, Decision Making, and Your Business (Modus Cooperandi Mememachine Series) [Kindle Edition]

Excel Spreadsheet for Hyperproductive Scrum Teams – very cool!

Cognitive Bias: Planning Fallacy

About the Author

Kevin has more than fourteen years commercial experience in the design and development of enterprise scale software applications. He is the author of both the Dynamic Range and Limited Days estimation techniques and is principal contributor to the IT Kanban Framework. He works with companies to support their transition to lean and agile methods from team to enterprise scale. Kevin provides independent consultancy and delivers courses in lean, Kanban, and agile practices, employing real world examples and experience to support organisations on their journey to continuous improvement. Recent clients include: Financial Times; Pearson; BAA; and Barclays.

33 Responses

  1. Jordan Mar 01, 2012 - Reply

    I agree it’s a misleading and sometimes vanity metric.

    However, I see the same in Cycle time; cycle time is the Lean/Kanban version of velocity, and cycle time even ignore estimation completely making it essentially irrelevant in the sense that if cycle time goes down, that is probably because the task was easier not that the efficiency has increased.

    Only when the cycle time for an identical task goes down would that metric have meaning and that is quite rare in software
    Jordan

    • Jim Benson Mar 05, 2012 - Reply

      Jordan,

      I totally get where you are coming from, but in this case the task itself is almost irrelevant.

      If I ask you how long it will take you to get a task done that took you 2 hours last week – you will say 2 hours or something less than 2 hours because you feel that you can beat your previous time with your new experience.

      But what you fail to take into account is the actual completion time of the two hour task – upon starting you might (will) be interrupted several times by unforeseen, yet predictable, events.

      Cycle time dispassionately measures the actual completion time of tasks. Changes in the completion times of individual like tasks rarely has much meaning unless you do those particular tasks many many times a day.

      However, changes in the flow of overall work is much more valuable for knowledge workers. Cycle time can show us, as one single aggregate indicator, how long it really takes to finish work in general. You can analyze that by work type, class, etc – but in the end you aren’t looking at individual changes in performance for individual actions. Knowledge work is too complex for those correlations to be drawn.

      • Jordan Mar 06, 2012 - Reply

        Modern day tasks are rarely equivalent…Facebook API one day…something else the next..cycle time is not useful unless you’re doing grind it out CRUD apps
        Jordan

  2. Jordan Mar 01, 2012 - Reply

    I agree it’s a misleading and sometimes vanity metric.

    However, I see the same in Cycle time; cycle time is the Lean/Kanban version of velocity, and cycle time even ignore estimation completely making it essentially irrelevant in the sense that if cycle time goes down, that is probably because the task was easier not that the efficiency has increased.

    Only when the cycle time for an identical task goes down would that metric have meaning and that is quite rare in software
    Jordan

    • Jim Benson Mar 05, 2012 - Reply

      Jordan,

      I totally get where you are coming from, but in this case the task itself is almost irrelevant.

      If I ask you how long it will take you to get a task done that took you 2 hours last week – you will say 2 hours or something less than 2 hours because you feel that you can beat your previous time with your new experience.

      But what you fail to take into account is the actual completion time of the two hour task – upon starting you might (will) be interrupted several times by unforeseen, yet predictable, events.

      Cycle time dispassionately measures the actual completion time of tasks. Changes in the completion times of individual like tasks rarely has much meaning unless you do those particular tasks many many times a day.

      However, changes in the flow of overall work is much more valuable for knowledge workers. Cycle time can show us, as one single aggregate indicator, how long it really takes to finish work in general. You can analyze that by work type, class, etc – but in the end you aren’t looking at individual changes in performance for individual actions. Knowledge work is too complex for those correlations to be drawn.

      • Jordan Mar 06, 2012 - Reply

        Modern day tasks are rarely equivalent…Facebook API one day…something else the next..cycle time is not useful unless you’re doing grind it out CRUD apps
        Jordan

  3. NickClare1 Mar 02, 2012 - Reply

    It can become a vanity metric if not used properly, but velocity should be a proxy measure of value delivered to stakeholders/customers/users. It’s useful at the team level but should always be ‘correlated’ against other, actionable metrics to make sure it isn’t just measuring activity.

  4. NickClare1 Mar 02, 2012 - Reply

    It can become a vanity metric if not used properly, but velocity should be a proxy measure of value delivered to stakeholders/customers/users. It’s useful at the team level but should always be ‘correlated’ against other, actionable metrics to make sure it isn’t just measuring activity.

  5. Steve Mar 03, 2012 - Reply

    All models are wrong – some are useful.
    Same is true of metrics

  6. Steve Mar 03, 2012 - Reply

    All models are wrong – some are useful.
    Same is true of metrics

  7. Lisa Crispin Mar 05, 2012 - Reply

    I’ve seen velocity so mis-used – for example, different teams w/in one company measured against each other in terms of velocity, as if that could be compared from team to team. My team never looks at it, tho one sprint last year, our SM noticed the pile of cards was 3x higher than average. That was from pushing biz to simplify.

  8. Lisa Crispin Mar 05, 2012 - Reply

    I’ve seen velocity so mis-used – for example, different teams w/in one company measured against each other in terms of velocity, as if that could be compared from team to team. My team never looks at it, tho one sprint last year, our SM noticed the pile of cards was 3x higher than average. That was from pushing biz to simplify.

  9. Baere Mar 05, 2012 - Reply

    I agree that humans are bad at estimating and I believe we need estimates anyway….

    In other words I don’t agree with the statement: “A reduction in remaining effort is of no direct value.”. Consider the following: a painter needs to do some work in your house. It’s the first time you see him, and he says he does not estimate the work as we humans are bad at estimating. Would you hire him? If you hired him and after a week – he seems halfway – you would ask what the remaining work is and he would answer ‘I have no idea as I can not predict the future’ what would you think of him?

    What about if you construct your house, would you like an estimate of when it would be finnished? Would you like an update of this estimation – or make one yourself – as the work gets along?

    I don’t see how replacing one metrix with another could solve something. It are just estimates so tread them like it. When I ask an estimate I typicaly ask for the order of magnitude. Is it 1hour/1 day/ 1 week/1 month? When asking how long it will take to build my house they don’t have to respond with an end date but with a number of months/years so I have an idea. The painter – in the example above – doesn’t have to say how many hours it will take as this is not feasable but the number of weeks should be doable. With keeping it as simple as possible nobody gets the false impression that estimations reflect reality.

  10. Baere Mar 05, 2012 - Reply

    I agree that humans are bad at estimating and I believe we need estimates anyway….

    In other words I don’t agree with the statement: “A reduction in remaining effort is of no direct value.”. Consider the following: a painter needs to do some work in your house. It’s the first time you see him, and he says he does not estimate the work as we humans are bad at estimating. Would you hire him? If you hired him and after a week – he seems halfway – you would ask what the remaining work is and he would answer ‘I have no idea as I can not predict the future’ what would you think of him?

    What about if you construct your house, would you like an estimate of when it would be finnished? Would you like an update of this estimation – or make one yourself – as the work gets along?

    I don’t see how replacing one metrix with another could solve something. It are just estimates so tread them like it. When I ask an estimate I typicaly ask for the order of magnitude. Is it 1hour/1 day/ 1 week/1 month? When asking how long it will take to build my house they don’t have to respond with an end date but with a number of months/years so I have an idea. The painter – in the example above – doesn’t have to say how many hours it will take as this is not feasable but the number of weeks should be doable. With keeping it as simple as possible nobody gets the false impression that estimations reflect reality.

  11. Baere Mar 06, 2012 - Reply

    I just checked out a couple of the links you mentioned in your article and it seams that the context in which we work/think is very different. Within the context of – for example – webside startup’s this is all very vallid. Actionalble metric are especialy important when you customers are independent from you. As you depent on the behavior of the customer It’s important to check it…Today I work in a big company and having actionable metrics for a technical SAP upgrade is not evident :-)

    Probably I have an issue with the fact that your statement “A reduction in remaining effort is of no direct value” is absoluut . For me a metric is context dependent and in some contexts the remaining effort has value. Especialy when there is not much incertainties like technical upgrades (new server, software upgrade…), legal updates or a client knowing exactly what he wants (yes they exists).

    Not sure what the metrix of this site is but you have a new reader – via a linkedin group – that commented on your article, checked out for more info and will come back ;-).

    • @jason – the variation in coding is precisely the reason to use cycle time. It is an aggregate statistical measure of real work actually completed.

      @baere- yes, all metrics are context dependent. One of the most important things about cycle time is the fact that in order to know what it is you need to have a good understanding of how your work is being done. That clarity into your work patterns is what lets you fully appreciate your context.

    • Kevin Ryan Mar 06, 2012 - Reply

      @baere – Great point regarding metrics being context dependent I couldn’t agree more with both yourself and Jim. I do believe that actionable metrics exist in scenarios that include technical upgrades and other aspects of life-cycle management. The question I would ask to discover what they are in these scenarios would be “why are we doing this, for what benefit?” This can often lead to more insight than purely a measure of how many activities we must complete before this is finished.

      And yes our own metrics certainly do include new readers and comments, metrics are people too, and as one of our new metrics you are most welcome and much appreciated. Thanks for both your input and involvement :)

  12. Baere Mar 06, 2012 - Reply

    I just checked out a couple of the links you mentioned in your article and it seams that the context in which we work/think is very different. Within the context of – for example – webside startup’s this is all very vallid. Actionalble metric are especialy important when you customers are independent from you. As you depent on the behavior of the customer It’s important to check it…Today I work in a big company and having actionable metrics for a technical SAP upgrade is not evident :-)

    Probably I have an issue with the fact that your statement “A reduction in remaining effort is of no direct value” is absoluut . For me a metric is context dependent and in some contexts the remaining effort has value. Especialy when there is not much incertainties like technical upgrades (new server, software upgrade…), legal updates or a client knowing exactly what he wants (yes they exists).

    Not sure what the metrix of this site is but you have a new reader – via a linkedin group – that commented on your article, checked out for more info and will come back ;-).

    • @jason – the variation in coding is precisely the reason to use cycle time. It is an aggregate statistical measure of real work actually completed.

      @baere- yes, all metrics are context dependent. One of the most important things about cycle time is the fact that in order to know what it is you need to have a good understanding of how your work is being done. That clarity into your work patterns is what lets you fully appreciate your context.

    • Kevin Ryan Mar 06, 2012 - Reply

      @baere – Great point regarding metrics being context dependent I couldn’t agree more with both yourself and Jim. I do believe that actionable metrics exist in scenarios that include technical upgrades and other aspects of life-cycle management. The question I would ask to discover what they are in these scenarios would be “why are we doing this, for what benefit?” This can often lead to more insight than purely a measure of how many activities we must complete before this is finished.

      And yes our own metrics certainly do include new readers and comments, metrics are people too, and as one of our new metrics you are most welcome and much appreciated. Thanks for both your input and involvement :)

  13. Mike DePaoli Mar 07, 2012 - Reply

    Lead Time and Velocity are both metrics that speak to predictability. I have my own experiences and hence opinions on these different metrics, but that’s not important as both can be used effectively in the right context. That said, neither of these metrics have anything to do with whether what is being predictably delivered is valuable to the customer and or user.

    This is where basic scientific method comes in to help us determine whether what we are building is valuable to customers / users. It teaches us to form an hypothesis about an outcome or future state, determine the learning that has to take place to validate our hypothesis, including how we will measure that learning and finally we design and build the experiment that we believe will result in the desired outcome. This is the inverted B-M-L cycle used during planning that Eric Ries refers to in The Lean Startup. I believe this pattern is a fundamental pattern that powers human curiosity. It’s the way baby’s learn as well as adults, but we don’t publish our metrics for others to see, they are encoded in our experiences.

    If you keep this basic pattern in mind and actually have effective testing of your value hypothesis with the intended recipients, actionable metrics are not that hard to identify or utilize.

  14. Mike DePaoli Mar 07, 2012 - Reply

    Lead Time and Velocity are both metrics that speak to predictability. I have my own experiences and hence opinions on these different metrics, but that’s not important as both can be used effectively in the right context. That said, neither of these metrics have anything to do with whether what is being predictably delivered is valuable to the customer and or user.

    This is where basic scientific method comes in to help us determine whether what we are building is valuable to customers / users. It teaches us to form an hypothesis about an outcome or future state, determine the learning that has to take place to validate our hypothesis, including how we will measure that learning and finally we design and build the experiment that we believe will result in the desired outcome. This is the inverted B-M-L cycle used during planning that Eric Ries refers to in The Lean Startup. I believe this pattern is a fundamental pattern that powers human curiosity. It’s the way baby’s learn as well as adults, but we don’t publish our metrics for others to see, they are encoded in our experiences.

    If you keep this basic pattern in mind and actually have effective testing of your value hypothesis with the intended recipients, actionable metrics are not that hard to identify or utilize.

  15. Hugh Mar 09, 2012 - Reply

    I totally disagree with Kevin and NickClare, above. Velocity was never intended to be “a measure of success.” If you’re using it that way, you’re using it wrong. Velocity is a measure of how much of the work you thought you had you actually got done, nothing more. Customer acceptance is the *only* measure of success.

    Also, it’s true that velocity is based on your estimate of the size of the task–but over enough tasks, a team’s estimate is consistent enough that the velocity is consistent and useful. Note that the estimate doesn’t even have to be accurate–just consistent, on average.

    • Jim Benson Mar 11, 2012 - Reply

      Hugh,

      Most people are using it wrong. A number, especially an integer, implies certainty. I know fully what velocity was intended for.

      However, the sprint commitment, when combined with velocity leaves people _expecting_ that certain amounts of work can be done in a fixed time. Given that the average is going to be low 50% of the time, the ability to hit your sprint commitments should also happen 50% of the time.

      I am totally comfortable with that.

      However, the “other” customer we don’t discuss is the rest of our business. Bosses, other teams, partners, who are counting on delivery of our product to enable their productivity. These people hear words like “commitment” and “velocity” and they interpret it as “predictability” and by “predictability” they mean “on-time guarantee.”

      This is natural and understandable on their part. They want to do quality work, just like we do.

      They don’t understand that velocity is based on story points which are completely relative numbers. They don’t understand that we will only make commitments 50% of the time in a sprint. They don’t understand that velocity is not a static performance guarantee.

      And, to make matters worse, these measures do not have value to them. So, if we treat any of these measures as a product – and the rest of our extended team as customers – the end user does not wish to buy what we are producing.

      Jim

  16. Hugh Mar 09, 2012 - Reply

    I totally disagree with Kevin and NickClare, above. Velocity was never intended to be “a measure of success.” If you’re using it that way, you’re using it wrong. Velocity is a measure of how much of the work you thought you had you actually got done, nothing more. Customer acceptance is the *only* measure of success.

    Also, it’s true that velocity is based on your estimate of the size of the task–but over enough tasks, a team’s estimate is consistent enough that the velocity is consistent and useful. Note that the estimate doesn’t even have to be accurate–just consistent, on average.

    • Jim Benson Mar 11, 2012 - Reply

      Hugh,

      Most people are using it wrong. A number, especially an integer, implies certainty. I know fully what velocity was intended for.

      However, the sprint commitment, when combined with velocity leaves people _expecting_ that certain amounts of work can be done in a fixed time. Given that the average is going to be low 50% of the time, the ability to hit your sprint commitments should also happen 50% of the time.

      I am totally comfortable with that.

      However, the “other” customer we don’t discuss is the rest of our business. Bosses, other teams, partners, who are counting on delivery of our product to enable their productivity. These people hear words like “commitment” and “velocity” and they interpret it as “predictability” and by “predictability” they mean “on-time guarantee.”

      This is natural and understandable on their part. They want to do quality work, just like we do.

      They don’t understand that velocity is based on story points which are completely relative numbers. They don’t understand that we will only make commitments 50% of the time in a sprint. They don’t understand that velocity is not a static performance guarantee.

      And, to make matters worse, these measures do not have value to them. So, if we treat any of these measures as a product – and the rest of our extended team as customers – the end user does not wish to buy what we are producing.

      Jim

  17. Jim Brisson Mar 12, 2012 - Reply

    Sorry, but I have to disagree with both points. I think this boils down to measuring Return on Investment (ROI) which is composed of value at a given cost.

    The product owner/customer determine what is done, and in what priority order, and so they have a very large stake in delivering business value. Accountability for doing the right thing rests on their shoulders. A|B EXPERIMENTs can be done when the product owner can’t decide which of two choices has more value, and so the experiment does not really reflect on the quality of their decisions (since they didn’t make the decision, they let the customer decide). A|B COMPARISONs of new features vs old could well be used to evaluate product owner decisions. Of course the comparisons, will not reflect the cost of feature development nor team productivity, so they don’t provide a full ROI metric …. and in my house, we always consider the cost when we buy.

    Return on investment measures value delivered for a given cost. Medium value at low cost may be preferable to high value at high cost. So the product owner needs the cost estimate to prioritize features appropriately. Of course there are many ways to get to the cost. But this article proposes throwing out velocity which is a key part of how agile projects typically determine expected cost, and therefore ROI.

    As per the wisdom of crowds, we can expect the velocity, which is a measure composed from multiple samples, to be about right (some samples high, some low, and some lucky). Yes they’re individual guesses, but the overall average should be about right. If you expect any individual guess to be right, then you’re using velocity incorrectly. Velocity allows you to see how much work a team accomplishes on average so that you can estimate how much work you can accomplish in the future.

    Burn downs show how much work remains and so is a reflection of work done by the development team. This should not be confused with measures of business value delivered which would be a measure of the product owner/client. The client can look at a burn down and know for example, that 1 of 5 requirements are not yet complete, or to turn it around, that 4 of 5 requirements are complete (use burn up if the client doesn’t get it). Since the client/product owner has prioritized these requirements, the progress of the development team should be extremely interesting.

  18. Jim Brisson Mar 12, 2012 - Reply

    Sorry, but I have to disagree with both points. I think this boils down to measuring Return on Investment (ROI) which is composed of value at a given cost.

    The product owner/customer determine what is done, and in what priority order, and so they have a very large stake in delivering business value. Accountability for doing the right thing rests on their shoulders. A|B EXPERIMENTs can be done when the product owner can’t decide which of two choices has more value, and so the experiment does not really reflect on the quality of their decisions (since they didn’t make the decision, they let the customer decide). A|B COMPARISONs of new features vs old could well be used to evaluate product owner decisions. Of course the comparisons, will not reflect the cost of feature development nor team productivity, so they don’t provide a full ROI metric …. and in my house, we always consider the cost when we buy.

    Return on investment measures value delivered for a given cost. Medium value at low cost may be preferable to high value at high cost. So the product owner needs the cost estimate to prioritize features appropriately. Of course there are many ways to get to the cost. But this article proposes throwing out velocity which is a key part of how agile projects typically determine expected cost, and therefore ROI.

    As per the wisdom of crowds, we can expect the velocity, which is a measure composed from multiple samples, to be about right (some samples high, some low, and some lucky). Yes they’re individual guesses, but the overall average should be about right. If you expect any individual guess to be right, then you’re using velocity incorrectly. Velocity allows you to see how much work a team accomplishes on average so that you can estimate how much work you can accomplish in the future.

    Burn downs show how much work remains and so is a reflection of work done by the development team. This should not be confused with measures of business value delivered which would be a measure of the product owner/client. The client can look at a burn down and know for example, that 1 of 5 requirements are not yet complete, or to turn it around, that 4 of 5 requirements are complete (use burn up if the client doesn’t get it). Since the client/product owner has prioritized these requirements, the progress of the development team should be extremely interesting.

  19. brent cowgill Mar 13, 2012 - Reply

    on my team I can think of a value metric immediately. we can ask the ad sales team for ad sales figures for our webapp. when we go live with new ad positions for the app, we can see whether those sales prospects have materialized and generated any actual revenue.

  20. brent cowgill Mar 13, 2012 - Reply

    on my team I can think of a value metric immediately. we can ask the ad sales team for ad sales figures for our webapp. when we go live with new ad positions for the app, we can see whether those sales prospects have materialized and generated any actual revenue.

  21. Piotr Trojanowski Mar 21, 2012 - Reply

    I must say this is a very interesting post. It represents an opinion that is quite common these days, I am afraid. Plus it is nicely wrapped into the guesswork-as-an-integer-value-slogan ;) Fortunately, the situation is not that bad in my opinion – one just needs to stop mixing apples with peaches in a single basket. Here are my thoughts:

    1. Estimation, velocity, time remaining, etc. are concepts that build one of a long list of statistical models aimed to predict work completion date. Usage of models throughout disciplines of science is indispensable. Still, when using a model one needs to keep in mind its nature and built-in limitations (e.g. one cannot apply classic physics to atom-scale phenomenon – it needs to be described by quantum model.).

    2. Estimation only becomes “guesswork described as an integer value” when someone, usually a development manager or sales department, grabs the estimates and turns these into a commitment expressed in number of days. Such approach clearly violates the core assumption behind statistical model predicting work completion date – estimations are not commitment, estimations represent current development team’s understanding and assessment of complexity of what is to be done.

    The proper question here is why let the bad guys turn estimations into commitment? Or in the wider context, the question is why have we been letting them do this for 30+ years?
    Agile is on the scene for 10+ years now, so it’s time to get out of the estimation-turned-into-commitment-behind-the-scenes ghetto. The true answer to meeting deadlines is variable scope!

    3. Activity metrics serves to feed current data into the statistical model aimed to predict work completion date. There is nothing wrong with it. On the other hand, actionable metrics remains in the domain of project owner’s accountability. Although this responsibility can be delegated to a develoment team, activity metrics is an independent topic having nothing to do with the statistical model of predicting work completion date. No reason to blame statistical models for not meeting business goals.

  22. Piotr Trojanowski Mar 21, 2012 - Reply

    I must say this is a very interesting post. It represents an opinion that is quite common these days, I am afraid. Plus it is nicely wrapped into the guesswork-as-an-integer-value-slogan ;) Fortunately, the situation is not that bad in my opinion – one just needs to stop mixing apples with peaches in a single basket. Here are my thoughts:

    1. Estimation, velocity, time remaining, etc. are concepts that build one of a long list of statistical models aimed to predict work completion date. Usage of models throughout disciplines of science is indispensable. Still, when using a model one needs to keep in mind its nature and built-in limitations (e.g. one cannot apply classic physics to atom-scale phenomenon – it needs to be described by quantum model.).

    2. Estimation only becomes “guesswork described as an integer value” when someone, usually a development manager or sales department, grabs the estimates and turns these into a commitment expressed in number of days. Such approach clearly violates the core assumption behind statistical model predicting work completion date – estimations are not commitment, estimations represent current development team’s understanding and assessment of complexity of what is to be done.

    The proper question here is why let the bad guys turn estimations into commitment? Or in the wider context, the question is why have we been letting them do this for 30+ years?
    Agile is on the scene for 10+ years now, so it’s time to get out of the estimation-turned-into-commitment-behind-the-scenes ghetto. The true answer to meeting deadlines is variable scope!

    3. Activity metrics serves to feed current data into the statistical model aimed to predict work completion date. There is nothing wrong with it. On the other hand, actionable metrics remains in the domain of project owner’s accountability. Although this responsibility can be delegated to a develoment team, activity metrics is an independent topic having nothing to do with the statistical model of predicting work completion date. No reason to blame statistical models for not meeting business goals.

  23. Halim Dunsky Nov 14, 2013 - Reply

    The continuing misunderstanding of estimates as commitments is a common situation that I refer to as “where are the grownups?”. This problem predates Agile. Sadly, the Scrum community has unintentionally contributed to it with the motivational concept of the sprint commitment (which, however, is no longer part of the Schwaber/Sutherland orthodoxy).

    In most large organizations and many smaller ones, schedule predictions are essential for many reasons, including ROI estimation (or its cousin, gut-feel prioritization) and the need to coordinate the work of multiple development teams, marketing, training, and so on. Both velocity and cycle time can serve here, but the more unpredictable the environment, the worse the predictions. All teams encounter some level of personnel variation over time, technology surprise, learning, outside interference, external delays, and so on. If the level of disruption is low and more or less steady, then both velocity and cycle time will reflect that and provide useful predictions on average, once there is sufficient historical data about the performance of the team in its workflow context. If the level of disruption is less predictable then the historical data becomes less relevant and its predictive value degrades.

    None of this has to do with measuring the delivered benefits generated by software in production. Once it goes online with users, how much value does it actually deliver? That critical input to product management is too frequently overlooked.

Leave a Reply