Software development, the Pareto Principle, and the 80% solution
I have spent the better part of a decade advocating the “80% solution” for software development projects. I talk about it a lot, as anyone on the Ricochet team can attest. I firmly believe that software, on the whole, costs much more than it should—and I believe that the Pareto Principle, or the 80/20 rule, as many know it, is the secret that can dramatically lower software development costs.
First, though, what is the Pareto Principle?
The Pareto Principle has its roots in a concept developed by Italian engineer, sociologist, and economist Vilfredo Pareto in the 1800s. Pareto observed that 80% of the property in Italy was owned by 20% of the citizens. Fast forward to the mid-1900s, when management consultant Joseph M. Juran, observing that about 80% of quality problems in manufacturing could be attributed to 20% of the causes, made the connection to Pareto’s work and coined the term “Pareto Principle” as a sort of universal law that could be applied just about everywhere.
For example:
- 80% of car accidents are caused by 20% of all drivers.
- 80% of taxes are paid by 20% of taxpayers.
- 80% of the vegetables in your garden come from 20% of the plants (believe it or not).
- 80% of many companies’ revenue comes from 20% of products and clients.
The list goes on—and I’m sure that if you put some thought into it, you can find many examples in your day-to-day life where this holds true. To be sure, it’s not always exactly 80/20. Sometimes it’s 90/10. Sometimes it’s 70/30. But, for the most part, results tend to fall into this simple Pareto Distribution.
As it pertains to software development, here are a few more examples that I’ve observed over the years:
- 80% of the bugs in software come from 20% of the features.
- 80% of the complexity in a given application comes from 20% of the code base.
- Only 20% of a given feature set in an application is important to 80% of the users.
- Conversely to the previous point, 80% of your users probably don’t care about 80% of the features in your application (they only care about 20%).
- A given engineering team spends 80% of its time on 20% of the application.
And here’s where this idea starts getting *very* powerful:
- 80% of your application (if you could get away with *only* doing this 80%) could be developed in 20% of the time and budget.
This last point is going to be the main thrust of this blog post. There are some caveats and dangers with allowing software to be developed utilizing this concept and principle. But my primary contention is that if you could address the points I’ll discuss below, you can develop software more quickly, less expensively, and with less head- and heartache.
But before I get into my argument, I want to make a few bold assertions about software engineers, and, really, people in general. I will argue that humans are fickle and prone to cognitive biases that produce unintended consequences. And that, on the whole, utilizing the paradigm I’m advocating will achieve better results than otherwise.
I want to share with you a few things I’ve observed while leading and managing software developers (and that I have personal experience with as an engineer myself):
- When software engineers are really excited about something, they can geek out on every aspect of it. This flow state is great, but can result in features that are over-engineered, with aspects that are more important to the engineer than they are to the client or business team requesting the feature.
- Software engineers often think long-term. They might build a feature so that it can last, regardless of whether it *needs* to last. For example, a proof-of-concept might literally be rewritten entirely once it is validated. Or it might be tossed entirely if a feature or application is unable to get traction.
- People who are not software engineers have a hard time understanding everything that goes into a given application or feature. This *can* lead to a lack of accountability for costs entailed in software development. Even for the most well-defined features, a software engineer makes assumptions that can cost serious time and money. Were the business team requesting the feature able to understand the implications of some of these assumptions, they would often opt for a simpler approach. But because of the way software is developed, these discussions often don’t happen.
- In general, no matter how hard we try, nearly everyone is pretty bad at communication. Walk away from even the most productive meeting and ask five people what was agreed to, and you’ll likely get five different answers. This miscommunication results in very real costs. This is also true for software engineers.
- It’s often safer to over-engineer than to under-engineer. If a critical bug makes its way into software, or an application crumbles under the load of a production environment, fingers get pointed at the engineering team.
- It’s often quite difficult to track costs to a particular feature, let alone particular elements of a given feature, in the software development process. Costs are more commonly looked at in aggregate, either as the overall “burn” of a project’s budget, or worse, as a sort of fixed cost on an organization’s budget (i.e. payroll). Even for the best teams, holding an engineer accountable for a dollar is extremely tough. Cutting costs to the level that I’m advocating requires a hunger to do so, and unfortunately, most organizations are simply don’t have the tooling to allow for this.
I’m certain that some will take exception to a few (if not all) of the points above. They aren’t intended to be categorical statements. But let’s just say that they are mostly true about 80% of the time ;)
So, let’s come back to my central point. If you *could* hold your team accountable on the feature level, and you could motivate them to identify opportunities to reduce costs by making potential trade-offs explicit, thereby letting the client or internal customer decide if additional costs are worth it, you could develop software more quickly, for less money.
Further, if the client or internal customer were to help the software developer understand what the “80% solution” might look like, the engineer would be a better steward of value and costs in the software development process.
If these things were possible, more conversations like the following would happen more frequently:
"Hey, business department, I know you asked for X—and this is certainly what I’m seeing implied by the designs you provided—but it’s resulting in some complex functionality in the sidebar that is going to take quite a bit of time to implement. If I could leave that bit out, I could get this done in two hours rather than 10. Is this extra bit of functionality worth eight hours of development time?"
In many instances, the additional bit of functionality is simply not worth it. And were it made explicit, everyone would agree. Multiply this by 1,000 over the life of a software project and you end up with some significant savings!
But what is the downside?
The main challenge with this approach is that sometimes, the 80% solution simply isn’t good enough, which is a valid point. To this, I would answer that I am not advocating a unilateral call to implement crappy software with corners being cut left and right. Rather, I am advocating *discussion* around where the tradeoff should be. Maybe the 87% solution is ok. Or the 92% solution. Shoot, maybe we all decide as a team that in this case, for this particular feature, the 100% solution is required. But either way, these decisions could be made as a team, in an environment in which both business and engineering get to weigh in.
To be fair, software engineers are often not the most gifted communicators, and engineering teams are frequently siloed from the people with whom the team must communicate to truly unlock value in this approach. But this is by no means an insurmountable problem if communication is made a priority.
Next, I have observed that the best way to achieve the value I advocate is to have *very clear requirements* for features being developed. This is really hard and requires discipline at many levels of the organization. It’s infinitely tempting to just lay out the gist of a feature and expect the engineers to understand and figure it out.
But when the project’s success is left to people “just figuring it out,” things invariably take longer, need to be redone, and are often implemented in a less than ideal manner. You can’t ever save costs because you have to keep redoing the work. And the fact that everyone (including the engineer) ends up frustrated doesn’t help matters.
To save costs on a software project, you need an environment in which everyone truly and clearly understands what is needed—as well as the value of the feature. When this is clear, and clearly communicated, software engineers can help laser in on the 20% that is *most* valuable and focus on that. They can also offer novel alternatives that can yield further savings. But when the environment is such that this isn’t possible (and again, it takes a lot of work by everyone, not just the software engineer), it’s very hard to yield any savings using the Pareto Principle.
All that said, estimation is notoriously hard. For any given feature, and especially when it’s particularly cutting-edge and tricky, it can be hard to truly see into the future and know how deep the “cave” is going to be (and if you don’t know anything about the cave, how do you even know how to prepare?). Every software engineer has had the experience of being blindsided by rabbit holes, deceptively complex features that you could never have suspected until you really got deep into the project—and it doesn’t help that even with features on which the team has spent considerable time to lend clarity, requirements sometimes change.
What to do?
In my experience, four practices can fortify against the worst effects of lack of communication between a business client and a team of developers:
- Peter Drucker, a famous management thinker is quoted as saying, “What gets measured gets managed.” At Project Ricochet, we simply must do this in order to bill our time, and the software we use allows us to track which feature we’re working on with incredible granularity. In order to set clear expectations, we also do an estimate in advance for every single thing that we do. When we first instrumented our systems to give our engineers visibility into their own personal “accuracy metric,” the numbers were all over the map.
But week after week, over the period of about a year, we were able to untangle the myriad reasons why (organization, personal, professional, client, and technical factors all played a part in it). Today, most members of the team come within 5% of their estimates (even when a different person estimated on the feature in question) about 97% of the time. I also have an average-adjusted metric in place that accounts for differences in the original estimate, which allows me to better coach engineers because I can see if they are under- or over-estimating to begin with.
By triangulating these two numbers along with a feature “reject rate” (the client rejected the ticket because issues needed to be addressed), I can tell if a software engineer is truly fast, or just simply slow.
Honestly, I geek out on this stuff and the above has taken years of painstaking effort, so my organization may not be reflective of the average one, but I contend that allowing people to estimate, and holding them accountable to the end estimate (Going over is ok! You just need to look at the time involved and know what the final result is) will make engineers better estimators over time.
- Next, software engineers need to realize that they are not spending their own money on a feature, and therefore need to alert the client or internal customer *as soon as* it’s clear that a feature is going to take longer than it was expected to—and provide an updated estimate for the remaining time.
The simple truth of the matter is that at some point, the feature might not even be worth it to the organization (in other words, “It was worth it at 10 hours, but not at 100 hours, and since we’re only 20 hours into that 100 hours, let’s stop where we are for now and come back to this if we have time at the end of the project”).
For this to work, engineers cannot be punished for being honest, or for underestimating. They need to feel comfortable sharing their honest assessment as stewards of the budget.
- And on the topic of communication, I have found that forcing teams to communicate on features above a given level of complexity yields great results. Not always, but often, software engineers want to just “jump into it” and get started. Instead, I advocate that any feature that will take more than a few hours of work requires the developer to first lay out a technical approach for the feature and get sign off by another engineer with domain experience in that area. This isn’t foolproof, but it is one more layer of defense against poor choices from the onset.
- Finally, software engineers need leadership to be the best it can be. A good manager can sense when an engineer is in an “ego trap” and possibly falling too much in love with his or her code, resulting in extra costs and time delays. Software developers are smart, and therefore are tricky to assess, but someone with experience knows the warning signs.
More importantly, though, good management lays the groundwork for the open and honest communication I advocate above. When talented people are able to try their best, and succeed or fail in a supporting environment, then you get the best out of them. Good managers help smooth over delays and overruns with internal customers or clients.
And they help hold software engineers accountable over time, providing resources, gut checks, and sometimes rude awakenings when needed (delivered with compassion). If you can, hire more of these people. They are rare!
Concluding thoughts
Honestly, I could talk about this topic for hours. I could give example after example of organizations leveraging (and not leveraging) this principle to dramatic effect. I’ve seen impossible projects get delivered successfully on shoestring budgets—as well as the opposite: relatively simple projects that dragged on and on. And on and on. And even then, didn’t quite hit the mark.
The reality is that the intangibility of software lends itself to a sort of inherent “squishiness” in the software development process. Couple that with cognitive biases that come baked into pretty much every human’s thought processes and you have a recipe for dysfunction.
To build truly great software, on time and under budget, you need at least two things:
- A truly engaged client or project champion, ready to put in the work to make sure that requirements are clear, deliverables validated, and miscommunications addressed as quickly as humanly possible.
- Software engineers who are truly committed to delivering the kernel of value in a given feature, and not afraid to offer smart alternatives to the explicit ask in the interest of the business value.
At Project Ricochet, we work hard to instill, foster, and cultivate the values that help our engineers succeed with point #2. And we work equally hard to help clients understand how to best actualize point #1 (in fact, please watch our recorded webinar about the Ricochet Discovery Process to learn how to really laser in on the value of your project—and present it in a format that is most easily digestible by software engineers).
If you’d like to talk more about how the 80/20 rule might help you or your team produce better software for less, please reach out. We’d love to hear from you!
Project Ricochet is a full-service digital agency specializing in Open Source.
Is there something we can help you or your team out with?
A bit about Casey:
