Observations on Software Estimation
During my career, I have worked in many IT businesses and, in that time, have delivered countless software projects, features, change requests and enhancements. A universal truism throughout has been that understanding how long something will take to build or how much it is likely to cost has been critical. The challenge is that this is incredibly hard to do, and often the people we ask are intimidated or worried about getting this wrong no matter how much we try to convince our engineering team they will not be held accountable. It doesn’t matter if you’re a services business looking to provide a customer with a quote or a product company looking to develop a new feature; getting this wrong can have severe and, in some cases, existential consequences. Deciding on building feature A instead of feature B depends partly on how long each will take to build, so for Product Managers, this is vital.
I know this is a familiar challenge for anyone working in software development, and I believe this is becoming a greater and greater problem as the cost of development climbs and budgets are under increasing pressure. Consequently, making a bad call will have more and more of a business impact.
I want to try and provide some high-level thoughts on estimation to provoke a dialogue. I look forward to hearing about how others have approached this and their thoughts on the topic. I’m genuinely interested in hearing what has worked for you.
If you’re a manager or executive trying to understand why this is hard, I hope there will be something here to take away. If you’re a developer looking for suggestions on how to get better at estimating, I hope you will find something here for you too.
Understand that estimations are estimates
Estimations are just that: they are estimates based on information available when asked. This means that everyone should expect these to change and evolve as time progresses. We would all like our estimates to be 100% accurate, but the reality is that this is almost impossible. What we can do, however is:
Monitor and track progress against those original estimates – too often, teams are asked to provide an estimate, but it is never revisited or changed when more data is available or outside influences change our working assumptions.
Alert stakeholders as soon as possible when we have deviations with an understanding of why that deviation has occurred.
Use data to inform future estimations (although teams rarely have the time or the data in a helpful form to accomplish this).
In a previous business developing eCommerce solutions, we encountered an existential issue: projects were overrunning, and we were at best breaking even on project implementations. The answer that turned things around and led to taking the business to a 30% EBIT tour de force was the creation of a model that encapsulated everything we learned about developing eCommerce solutions. Any future project, change request or enhancement was fed through the model to produce complete estimations, costings, resource and project plans. We were unbeatable at that time as we could provide fixed cost plans in minutes as opposed to our competition, who could only offer “time and material” estimates after days and weeks of analysis. Any new insight we gained resulted in a model change to maintain our 30% EBIT.
Don't overthink it
Overthinking is the downfall of estimation. It takes many forms, including:
Inability to provide an estimate until a proof-of-concept has been created.
Inability to provide an estimate until a deep analysis lasting many weeks has been performed.
Inability to provide an estimate until every team member has been asked their opinion.
I suggest each of these issues is cultural and often stems from fear. We have established that estimates are estimations and should evolve once we have more information. With that in mind, we should be able to provide a high-level estimate early with the comfort of knowing that it can and will change. There are situations where there is validity in developing a spike (short investigation) to inform an estimate or seeking additional assistance. Still, it should not be the norm and default behaviour to always look for external validation and delay communicating a number. It’s better to provide an estimate and develop a culture of making an educated assessment early than waste lots of time and effort coming up with an arguably, similarly accurate estimate later. I would suggest that this way of working impacts “flow of work” and causes untold issues and frustration.
This overthinking can and does cripple an organisation. To the stakeholder, it holds up decision-making. To the engineer, it causes stress and anxiety. Ultimately this causes friction, and a negative culture ensues.
Consider that a stakeholder told to wait for an estimate until more “analysis is done” provides the stakeholder with legitimacy that the number finally provided is super accurate because the estimates provided have been painstakingly scrutinised. In reality, the estimate following that analysis is still an estimate. A better approach is to give an estimate with some caveats and uncertainties, perhaps even a range, and commit to improving it over time. However, the organisation’s culture needs to support this approach to avoid any negative impacts.
Also, consider that an executive or stakeholder who chastises a team or individual (even privately) for not getting the estimate 100% correct reinforces the behaviour that caused the problem in the first place. It’s a very unvirtuous circle that needs to be avoided at all cost.
Experience matters
Of course, it does. If someone has implemented a similar feature in the past, they will likely understand better how long something will take to build. However, we often rely on that information being in an individual’s head. Understanding and having expertise in estimation is just as much a piece of intellectual property as knowing how to develop and engineer the solution itself. As an industry, however we are very bad at capturing this form of intellectual property and rely more on memory which can be a flawed device.
Experience, therefore, is not just about personal experience but also about capturing past expertise in a form that can be used for future benefit and future estimation. Logging original estimates, tracking their accuracy and then actively utilising this information for future estimation purposes should be done, but tools to support this activity are few and far between.
Trust is important
There are environmental factors that impact estimations, including trust. This comes down to culture, but there needs to be trust between the engineering team and the wider business. I have experienced friction and conflict being encouraged to create tension between different departments. Whereas I wholly subscribe to a sense of urgency and commitment to dates being important (I call this professional pride), this tension can boil over, which may lead to the hiding of information, for example. Interestingly if the culture is wrong, this is all done subconsciously and at multiple levels within an organisation. For instance, the engineering team knows they are behind the ideal schedule but don’t want to actively communicate this as they know that another department would take advantage of this slippage for their own political end.
Much better is for all departments to be aligned and educated on how software is developed and the complexities it brings. That way, when a delay is announced, people band together, replan and do this collaboratively.
Most mature teams understand this and break up their implementation into sensible phases so that if the whole solution cannot be created within a specific timeframe, at least some smaller feature set can be released. Better still, the organisation recognises that to develop great software you build on it and deliver in small incremental steps garnering customer feedback along the way rather than attempting a big-bang approach. Estimates are still crucial when developing in this way – we need them no matter which approach is taken.
When it comes to estimation, trust is about being able to make incorrect estimates and not feeling like this will be a career-ending event when this happens.
It is no wonder that an engineering department may be reluctant to commit to a date or provide estimates if they are turned into pariahs each time they communicate a date that is seen to be missed.
However, this behaviour is doubly compounding as it prevents the engineering team from getting better at estimations. Instead of honing this crucial skill, they are encouraged to provide wildly outrageous estimates to avoid being seen as wrong. This scenario of over-estimating may then result in a business decision to be taken with incorrect information, such as to prevent a project from being started because the estimates are too high. What is really happening, however, is that the engineering team are just covering their behinds because the business culture and lack of trust has created that behaviour.
The final nail in the coffin is where stakeholders who suspect that estimates are being inflated secretly assume this and bring forward critical dates such as a marketing event, thus reinforcing the mistrust, and the organisation’s culture suffers further.
It is critical then that regarding estimations, there is trust that people are doing their best and also that there is joint ownership when it comes to hitting targets and joint accountability, encouraging all teams to work together. This encourages total transparency and understanding. When people understand how things work it’s amazing what contributions they can bring to every challenge you face. If this is not the culture that prevails, then counter-behaviours are formed that ultimately create a toxic environment and prevent teams from improving their estimation skills.
Estimate composition
We’ve covered some parts of estimation but not what is in an estimate itself. I have deliberately not ventured into the debate about time versus story points or other such units of measure and will not dwell on it here. Estimate composition though, that is, what’s in an estimate, is a critical piece that needs addressing as we must compare “apples with apples” at every stage in our process.
We must be very clear with ourselves and the teams being asked for estimates on which part of the solution you are asking them to consider. The full lifecycle of “development” from conceptualisation through to support needs to be assessed in the wider planning phase but not necessarily by the Product or Engineering teams alone. If I were to ask multiple teams to provide estimates for the same feature but not be clear about what I was asking the team to estimate, I would undoubtedly get wildly different answers. Team A may provide estimates based on just the development phase and getting the solution working in a test environment. When providing their estimates, Team B also considered testing and deployment into production. Team C considered all of this but forgot to factor in the support team’s handover and training.
It always comes as a surprise when, at the end of a development where teams say that the work is “done”, just how much remains to actually do before value is derived from that hard work and the new feature is ready to be enjoyed by your customers.
When estimating, it maybe better to keep things simple and only ask engineers to focus purely on the implementation of the core feature, its unit testing and the creation of any deployment scripts. All of the other aspects, such as regression testing, performance testing, technical documentation, support handovers, training, hyper-care support, production of marketing materials etc. can be considered separately. If you break things into stages like this, things seem to become more manageable.
This ultimately comes down to your team’s “definition of done” and the deployment processes, but it’s an important concept that is often missed and needs careful clarification from the offset.
Changing your “definition of done” or something fundamental can also impact future estimations. Suppose you include a new step such as the production of marketing materials into your workflow. In that case, you need to factor this into future calculations when reviewing similar work from the past and maybe add extra time. This is why logging your “definition of done”, and any other assumptions alongside your past metrics is imperative.
Impediments and unplanned work
Impediments and unplanned work are those things that keep getting in the way. When people start with estimations, they often assume complete dedication to that work. They assume that a person or a team will be unencumbered and are working 100% to deliver that work to hit the estimate provided. I think that is the right thing to do; otherwise, you are producing an estimate based on countless what-if scenarios that make comparing apples with apples impossible. You are, after all, creating an estimate for the effort required to complete a task all-other-things-being-equal.
When it comes to execution however, life is not always that simple and some things often get in the way. Organisations need to appreciate this and accept that distractions occur. Tracking progress and logging distractions will help manage your team’s velocity and hopefully help remove any unnecessary burdens that have been unintentionally placed on them. These things naturally happen over time and when organisations grow. Keeping a conscious eye on these things to improve overall efficiency is healthy and encouraged, which is why we need to log and track them in a simple and non-cumbersome way (non-cumbersome being the operative phrase). It is vital because senior leadership must accept that impediments and unplanned work impact progress and shift assumed dates. We can choose to remove these distractions so that the main focus of work can be completed unabated or accept a delay.
Teams themselves also need to appreciate what these delays are and come up with explicit processes to deal with them. This type of unplanned work is insidious as it creeps up on teams, and they often fall into a pattern of acceptance rather than questioning if things could be done in a better way.
Sometimes this leads those estimating, for example, to assume only a 70% available capacity for work. This results in larger and larger estimates, as we have seen before. When asked where the remaining 30% goes, the answer is typically “distractions”, to which senior management cry foul, demand proof, and say it’s “impossible”, leading to the unvirtuous trust cycle all over again. I am a proponent of “measuring what matters” and so-called distractions matter.
Example unplanned work and distractions include (reader insert your own here): company meetings and “all hands” events, support requests, requests for help from other teams, sales demos, customer meetings, production issues, and yes, even creating more estimates for future phases and development. Distractions are not bad things and we need them to function as an organisation but let’s not underestimate their impact.
Another issue with distractions or unplanned work is that it impacts focus. Each time a distraction occurs, it has a larger impact than imagined as somebody has context-shifted what they were doing, and it will take time to refocus on the work they were doing previously. If distractions are constant, then assume very little productive work can be achieved.
Every organisation needs to accept that these things happen, decide what level of effort should be considered acceptable for these activities, and factor these into planning. Raw estimation should be regarded as separate from planning, and distractions should not be factored into pure estimation numbers. I say this because the planner and the estimator roles should be separated. My definition here is that an estimator is that person estimating tasks whereas a planner is someone who takes those estimates and prepares a plan from them (yes they can be the same person, but the activity of estimation and planning is distinct). If the estimator factors impediments and unplanned work into his estimates, they can be double-counted when the planner takes those estimates and collates them when an overall plan is prepared. The planner role understands what can get in the way, how interdependencies impact raw estimates and what this translates to in terms of a number to commit. Don’t put that on the engineer who is thinking about their technical task, or the technical author who is writing the user guide. You will get better and more transparent plans if you separate the two roles.
Understand who will be doing the work
Estimations are further complicated by considerations of who will be executing the work. The junior developer fresh in the company will probably have a different implementation time than the seasoned engineer who wrote half of your codebase. This is a fact that you cannot get away from and it’s also a fact that you cannot assume your top developer will be available to implement all of the complex code you want them to work on all of the time.
The long term solution to this is to remove this skill discrepancy within the team and to have established personal development plans in place but that takes time to achieve and is a conscious effort that your organisation needs to make happen as it rarely happens by itself.
When estimating, one approach is to estimate based on averages. The question that needs to be asked is: “How long would it take the average team member to implement?”. That way, the averages should balance out when it comes to your Sprint planning or your resource planning process. You will win on some tasks and lose on others, but the extremes will cancel each other out, and your broader estimates should hold true.
It’s important to be consistent when you are looking at estimates and if you start estimating with individuals in mind, it makes historical estimation review challenging to achieve which is why trying to think of the average effort for a team member to complete as the sensible approach to take.
Optimism and pessimism
Personalities also factor into estimation. The optimist will typically assume the best, whereas the pessimist will typically think the worst. This can be very frustrating for an observer asking each personality type to provide the same estimate. The answer, of course, is somewhere in between which is why it can be useful to pair up opposites when looking at estimations. If trust has been established, then the two extremes should be able to consider each other’s perspectives and come up with a number they are both comfortable with.
At the very least, you should look to have a couple of people look at the estimates together. I don’t personally subscribe to every team member having a vote. Whereas I see the logic that one team member may have thought of something everyone else has discounted, my experience of estimation-by-committee is that lots of time can be spent collectively doing something that one or two could do in isolation very effectively. The impact of committee-based estimation is often a reduced overall team velocity as some of those team members could be working on technical tasks instead of, well, estimating.
A way of asking the wider team to validate estimates prepared by others can be far better than asking each team member to come up with the estimates themselves in a voting fashion which is more efficient and can achieve the same result. Consider rotating this responsibility over time so that everyone gets an opportunity to flex their estimation “grey matter”.
In conclusion
I subscribe that software estimation and review is a critical skill that is often overlooked and we need to help train our teams to develop this competency. It is a multi-dimensional challenge and in this post, I’ve summarised some factors:
Develop a trust-first collaborative culture as a first step to encourage experimentation and estimation processes. Without this environment estimations will be considered a poisonous exercise that nobody wishes to participate in.
Estimation is a skill that needs to be developed and is as essential as any other we covet, such as proficiency in a particular programming language or speaking effectively in public. It is very much an underestimated skill (pun intended), but I believe it needs to be given more prominence and dedication.
Organisational culture has an impact, and all parties must be accountable for how people treat estimations. We will get these estimates wrong more times than we get them right, at least initially, and that’s okay. The critical part is making an effort to try. Developing a culture of trust and understanding will make teams better at estimating much quicker and lead to longer-term super-estimators.
Estimation and planning are separate activities. When estimating, we must ask our team to focus on the “definition of done” part of estimating, with all other things being equal and considering no distractions. Planning should take this estimation and make it real with real-world constraints and real-world limitations.
Logging and recording distractions (at least the magnitude of the distractions) will help with planning and hopefully lead to organisational change if the level of these distractions is not deemed acceptable.
Logging and recording how accurate our past estimates were will help with future estimations.
Software estimation can be a contentious issue, and I’ve tried to capture some of my thoughts and high-level observations on the topic to spark a debate if nothing else.
At Stratagems, we are developing the next generation estimation and planning platform for software businesses that allow teams to be better predictors. Our goal is to do all of this in the lowest-impact and most simplistic way possible: and maybe to even make it enjoyable.
We look forward to hearing what people think.
