Why We Stopped Estimating in Points

There has been a lot of debate and discussion around the practice of using story points for estimation in software development, so I thought I’d share our experiences at Moonpig and explain why we stooped using points.

This post was originally written for the Moonpig Engineering blog.

Shortly after our web teams at Moonpig transitioned from the Scrum framework to Kanban, we decided to stop estimating in points. As there has been a lot of debate and discussion around the practice of estimation, I thought I’d share our experiences and explain why we changed.

What is the point of points?

Before talking about why points are not always useful, it’s worth revisiting the “point of points”. Software development is notoriously unpredictable and it’s virtually impossible to accurately estimate how long a given piece of work will take. Points attempt to bring some predictability around delivery. If we break down a feature in to several chunks, give each chunk a relative estimate, and then compare the total estimate with the team’s velocity, we should be able to predict how long it will take to complete. Simples right?

In theory, yes. In practice it’s not nearly so straightforward. To provide reasonable estimates, your feature or project needs to be very thoroughly scoped with absolutely every use case captured. That’s tough to do. More use cases tend to emerge once you get started, so inevitably the scope of the project ends up being bigger than what you originally estimated.

Moreover, in order to provide estimates a certain amount of assumptions will need to be made and many of these will be wrong. There will be unknown, unpredictable factors. The more information you have, the more accurate your estimates. But the only way to really capture all the information you need to provide accurate estimates is to actually do the work.

So points based estimates are literally that — an estimate. And very often those estimates will be wrong. It requires a fair bit of time and effort to break down and estimate a project, and I’d question whether it’s time well spent. In my experience, you can get an equally “reliable” estimate simply by asking the team roughly how long they think something will take — 3 weeks? 6 weeks? 6 months?

The bad news is there really is no way to accurately predict when a piece of software will be completed. I wholly respect the thinking behind the points system; unfortunately it is often misunderstood and misinterpreted by those who are fixated with delivery dates. That a specific technique has underpinned the estimation unwittingly means there is far more confidence in it than there should be — it is after all an estimation, and that quickly gets forgotten. Estimation is not so much planning poker as Russian roulette!

Focus on Outcomes

In an environment where delivery is critical, knowing when something will be “done” is understandably valuable. However, if you work in a lean environment, you are only done when you have solved a customer problem. In these circumstances what matters is not output, but outcomes. A project is only complete when a customer problem has been solved. When we begin solving a customer problem we expect to experiment to find out what solutions customers want. We want to invest the minimum effort in these experiments, and only start to invest real time and effort once we have identified a viable solution. In this context a project is not a feature, but a problem to be solved. Delivering projects “on time and on budget” is of little value if your customers don’t like your feature or product. So in the lean world, using points is of limited value because when something is delivered is far less important than whether or not it will be successful

Cycle Time vs Velocity

Points can also be popular in order to measure velocity and track improvements. However, velocity, like points, is often misunderstood and consequently can be dangerous. Comparing velocities of different teams, for example, is futile. Velocity also shifts emphasis to the quantity of work delivered rather than the value delivered.

A much more helpful measure of team health and efficiency is cycle time. We want to solve customer problems quickly with the minimum effort; in this context tracking the time it takes you to get your ideas in front of your customers is far more valuable. Moreover, cycle time can accommodate the entire delivery cycle — from concept to customer — whereas points focuses purely on the development effort. To really achieve an efficient workflow, every element of the value stream needs to be optimised, not merely the software development portion of it.

Size matters

So have we stopped estimating altogether? The answer is no. Teams that no longer use points instead do a rough time estimate. There are two reasons for this

• Firstly, we want to keep each piece of work small. Breaking work down small is fundamental to maintaining efficient workflow. Estimating helps us ensure that we are doing that. Some teams will estimate the number of days they expect a piece of work to take — this is not a commitment, it’s purely to assess the size. Other teams will simply ask if the work can be done in 5 days or less. If the answer is no, they will look to break it down smaller.
• Secondly, getting the team to estimate the work ensures that everyone has clearly understood what needs to be done. If estimates vary wildly, it’s an indicator that the story needs more discussion.

Are points pointless?

I personally have no strong feelings on this either way. I think estimation should be at the discretion of the team. Our iOS and Android teams use the Scrum framework, and they find points incredibly helpful in planning sprints — which indeed they are.

I certainly don’t think using points should be mandatory, and if you do use them, use them wisely. These are my suggested do’s and don’ts:

Do

• Use points to plan sprints if you find that helpful
• Do use points to keep story sizes small if you’re not comfortable using time-based estimates

Don’t

• Rely on points to measure team health or efficiency — cycle time is a much better metric
• Use points as a reporting metric — points and velocity are misunderstood. Keep them for use within the team, but don’t share them. Shift the emphasis to reporting cycle time and outcomes the teams have achieved.

Originally published at engineering.moonpig.com on May 10, 2018.