When you think about improving software quality, your initial thoughts might be the cost of investing in new tools and engineering labor to implement them. You may also think about adding new processes to the software development cycle to improve quality. However, did you ever think of first focusing on what the cost to your company is to find and fix bugs throughout the product life cycle? Beyond the above direct costs, have you considered the opportunity cost associated with missed release dates or the negative impact on your brand resulting from unhappy customers or product recalls?
Quantify the cost of fixing bugs
In the book “How Google Tests Software – Help me test like Google,” the authors talk about the evolution of Google’s software testing process. The issue of bugs from incomplete testing was one of the biggest roadblocks to Google’s success. Google was able to quantify the cost of poor software quality in terms of the number of engineers involved in fixing bugs. As a result, they put a software quality improvement plan in place that included testing earlier in the software development life cycle and doing continuous testing. This freed up the same engineers to design new features and beat competitors to market. Below is an example of the calculation for a mid-sized project, on an annual basis, using Google’s numbers.
| Source Lines of Code (KSLOC) Generated Per Year | 200 | |
| Average Bugs Per 1000 SLOC | x | 8 |
| Number of Bugs in Code | = | 1600 |
| Average Cost to Fix a Bug | x | $1,500 |
| Total Yearly Cost of Bug Fixing | = | $2,400,000 |
| Year Cost of an Engineer | / | $150,000 |
| Number of Engineers Consumed with Bug Fixing | = | 16 |
| Engineering Team Size | / | 40 |
| Percentage of Staff Used for Bug Fixing | 40% |
Use the table above as template to calculate with your own numbers to estimate costs.
It’s likely that when you go through this exercise, you will find that your team is spending 30 to 50% of their time fixing bugs, and that’s just the direct cost. Imagine if you could use more of that time to create new features for your product? Furthermore, what if you can reduce your support costs because fewer bugs are getting to customers? Think about happier customers who purchase more of your products!
Improve your test process maturity to prevent bugs
Companies like Google, who successfully use a continuous software quality approach, have added automated testing to their development cycle, and it pays huge dividends. A key component to Google’s approach is a test platform to ensure that bugs are found as early in the software development life cycle as possible – when bugs cost less to fix.
The table below contains sample costs from Google of bugs found in different phases of testing, and it clearly shows that finding bugs earlier in the development process translates to a lower cost per bug.
| Software Testing Phase Where Bugs Were Found | Estimated Cost per Bug |
| System Testing | $5,000 |
| Integration Testing | $500 |
| Full Build | $50 |
| Unit Testing/Test-Driven Development | $5 |
Now that you have a better idea of what bug fixing costs your team, what is the best way to reduce bugs and improve quality? Well, the average cost to find and fix a bug is not going to go down over time. However, improving and automating your testing process will reduce your overall maintenance cost as shown in the figure below.
Companies find themselves at different levels of the spectrum. For example, some may have no formalized testing infrastructure while others, such as Google, have already solved this problem. Your company may find itself somewhere between these two extremes. Where should you start to improve software quality? As with most engineering tasks, a measured, step by step process is the most effective way to improve. Each step should build upon previous improvements as shown in the following figure.
Conclusion
Companies that adopt and embrace the value of bug prevention, instead of bug fixing, free up valuable engineering resources for product innovation and new product development. As a result, this enables them to gain a greater market share in the industries they serve.
A recommended starting point is to introduce code coverage and static analysis tools. These tools can be implemented without changing the workflow of your development team but will yield actionable metrics to help improve software quality. For example, System Test Automation enables organizations to generate code coverage metrics during any type of software testing and reports on which portions of your source code have been tested. In addition, it reports on which portions of your source code have not been tested.
7 thoughts on “Cost of fixing vs. preventing bugs”
Thanks Lynda, I enjoyed reading your article + the handy links at the bottom. Kurt
Testing is important, but you are missing a point.
You said ” … bug prevention, instead of bug fixing, free up valuable engineering resources for product innovation and new product development”
That is true, but then you said:
“A recommended starting point is to introduce code coverage and static analysis tools”
None of the above help prevent bugs. They can only detect them after they exist.
Your post is good, but understand that bugs, like any quality factor, are prevented from inception, design, coding practices, people management, leadership…
Testing is only one part of it, arguably, not the main tool because it can only be executed after the fact.
Quoting E. Deming:
“Cease dependence on inspection to achieve quality. Eliminate the need for inspection on a mass basis by building quality into the product in the first place.”
https://www.ifm.eng.cam.ac.uk/research/dstools/demings-14-points/
Alex-
Very good points indeed. There are definitely situations where quality “after the fact” makes sense. One is that there are large, legacy code bases out there that were not built with quality in mind. And now the owners of that code what it to pass a safety certification. Testing is the only way to get there as the design and coding are already done. And in fact the strategies outlined in the article probably are best used in this situation.
Also, once you have developed a high quality piece of software, and you want to add to it, it can be harder to maintain that high level of quality, thus incurring a lot of technical debt. So testing can be a critical piece in ensuring that something that used to work is still working after a software change is made. Keeping the architecture in sync with the code is also another good strategy, and there are tools for that as well.
Thanks for your interest in this topic!
Lynda
I could not find the information in Table 1: Metrics from Google in the book “How Google Tests Software – Help me test like Google”. Could you please direct me to where you exactly have this information from?
Thanks for reaching out Sebastian! There is no one place in the book that these metrics come from. The basic story that Google tells in the book has numbers here and there, and those were collected and put into the table. I hope that is helpful.
Thanks – that clarifies it 🙂