The Cost of Not Doing Testing
One of the maxims in software testing is that “testing early and testing often” helps avoid the high cost of deploying a software bug. But it’s hard to quantify what that exact cost is because of all of the factors that go into the software development life cycle, including (but not limited to) an organization’s size, the number of customers, and the employees and resources utilized to debug and fix. There are, however, some statistics that we can look at to determine how damaging bugs can be.
From a big-picture perspective, Tricentis (a software testing consultancy from the Bay Area) published a report with aggregated data from several industries about its software failures in 2017, and the dollar amount presented as the total loss is quite staggering: $1.7 trillion. The time value for these bugs is also enormous: the combined downtime for all the systems included is over 260 years. The industries most affected were retail and the consumer market.
But what’s happening on a smaller scale? What does that have to do with your specific organization?
While it might not be apparent at first, every bug that slips through the cracks grows exponentially in cost depending on when it is detected. The rework to make it go away adds up really fast.
Bugs detected during design phases (before any implementation is done) reduce the cost of fixing in a 5x proportion compared to bugs found during implementation (according to IBM’s research on its own software labs); bugs become 100x more costly if the error manages to go all the way to production and is detected during maintenance.
The further along you move into the end phase of feature/product delivery, the bigger the monetary impact you will have for fixing said error, as that implies rework on top of the initial implementation and the opportunity cost of the resources devoted to fixing it rather than working on new features or upgrades.
At this point, it might seem that errors need to be catastrophic to be detrimental to the operation of your organization, but that is never the case, as small bugs add up fast and, in turn, eat up the development budget.
Another issue that’s a high intangible cost is the detriment to brand perception and customer loyalty when errors manage to pass undetected all the way to the customer’s hands. Six years ago, when Apple decided to decouple itself from Google services as much as possible and launch its own version of a Maps service for iPhone users, it pushed a very buggy version of the app alongside the iOS 6 update.
The Apple Maps errors that sometimes duplicated locations, erased cities from the map, and sent customers on wild goose chases just to find a movie theater led to a trend in social media about the impending doom of the entire Apple ecosystem, #ios6apocalypse. That, of course, didn’t happen, but it was unquestionable that the brand suffered from the blowback and had to dedicate more resources to push fixes for the app as soon as possible. Being on damage control while also being in the spotlight is a very uncomfortable situation to be in.
What can be done to avoid these costs?
One thing for certain is that the only system that doesn’t have a risk of introducing bugs is a system that is never touched, and that is practically an impossibility, as business needs grow and change in every industry continuously. So the focus is on reducing risk (and, inherently, costs) rather than on eliminating it altogether.
One excellent way of reducing and mitigating risk is to introduce testing into the development process as an integral part of it instead of as an addendum.
Testing can happen in a manual or automated manner, and the recommendation for which one to do depends on the size and complexity of the project at hand. Is it cost-effective to always use automation from the beginning? Not really, because as the test suites and test sets tend to become a software entity of their own, they will also need maintenance. But if the project size and complexity warrant it, the ROI is rather quick.
Internally, we have detected that some projects that decide not to include testing because of budget constraints tend to have a higher rework rate, not because the development is lacking but because there is a missing bridge in the process between the technical and the business sides that sometimes helps clarify requirements so that they are better implemented.
Here at Project Ricochet, we advocate two major things: develop and iterate fast to deliver value, and test at every step of the way. Switching to Agile methodologies has helped us reduce feature rework rates on the projects that adhere to our process, because we are constantly testing and finding issues earlier.
The incremental approach allows for more feedback and flexibility, and every time a change is introduced into the code, it is checked for bugs. While at first it might seem that adding testing and QA procedures to the process will increase the monetary cost of any given project, in reality, the projects that do this as early as possible are the ones that usually come in under budget and before the deadline, because we are able to catch errors as early as in the requirements-gathering phase (the discovery process), modify the requirements to avoid the pitfalls, and adapt the implementation work with that mindset already in place.
Having a continuous delivery strategy regardless of the platform and technology stack we use for the project helps us iterate faster so that the testing doesn’t stop. That, in turn, helps the testing resources be more thorough in regard to the coverage of the tests they are performing.
So am I saving money by not including testing in my development project?
All evidence points to the answer being no. While it’s not possible to find every single bug out there, having a testing process throughout the development always mitigates the risk of high-impact bugs making their way to your production instances. That means fewer resources dedicated to fixing said bugs.
Even if you are just doing unit testing of a developer’s code, testing reduces risk exponentially compared to not doing it all.
There are several ways to integrate a testing resource or team into your development process if you don’t already have one in place—or you can use a development agency that has testing in its DNA to help you deliver a better product.
We’re passionate about these subjects, so let us take a peek into your idea or project, and we’ll help you bring it to life.
Project Ricochet is a full-service digital agency specializing in Open Source.
Is there something we can help you or your team out with?
