Sample Defect Cost Calculation
The cost calculation is not very simple and a lot of assumptions, and estimations are needed. The following definitions, terms, and formulas are a first step for an understanding of the costs related to defects and testing. Especially, the results shall be an example and maybe an argument for a strong and tight testing.
Let us define first the Defect Detection Percentage (DDP):
DDP [$] = (bugs detected) / (bugs present) = (test bugs) / (test bugs + production bugs)
The defect detection percentage is a quite simple formula, because it is just the ratio of the bugs found before delivery to the total bugs in the code which is unknown. The real insight comes, as soon as the ratio is rewritten to the ratio of the bugs found during testing and the sum of the found bugs and the bugs which are delivered into production. This formula shows in a very simple manner the simple truth: All bugs which are not found during test go directly into production. The first ratio is to abstract to get this point straight with the right emphasis, but that is really important: Bugs in production at customer’s site destroy reputation, decrease future revenue due to lost customers, and in worst case bring in calls for liability.
The relationship of total found bugs to critical bugs found should be under all circumstances:
DDP (all bugs) < DDP(critical bugs)
Or in other words: It should be the final goal to have a much better detection percentage for critical bugs than for all bugs or minor bugs. It is in most cases not very critical to have a button in a UI at the wrong position, but a crashing application or an application which cannot keep a database connection open, is critical.
The consequence is: We should put more time and effort into testing of critical and blocking defects than on minor issue testing. I have seen this in real world already. The focus and the effort on the UI was remarkable, because it was obvious and everybody could see it (especially the own management and the customers management), but the performance was not sufficient and the stability was not as good as needed. But, the product was shiny and looked beautiful…
Have a look into the post about the quality characteristics: http://rickrainerludwig.wordpress.com/2014/02/10/software-quality-charateristics-in-iso-9126. What is critical and what not may change from project to project. A deep though is needed for the weighting, but the weighting should lead the testing priorities.
The possible costs for defects (see Beautiful Testing: http://astore.amazon.de/pureso-21/detail/B002TWIVOY) can be classified as:
- Cost of detection: All costs for testing and accompanying actions. Even if no bugs are found, there are costs which need to be taken into account. For example: Performing a quality risk analysis, setting up the test environment, and creating test data are activities to be taken into account. They all incur the costs of detection and there are much more tasks to do. These are the permanent costs on timely basis.
- Cost of internal failure: The testing and development costs which incur purely because bugs are found. For example filing bug reports, adding new tests for the found bugs, fixing bugs, confirmation test bug fixes, and regression testing are activities that incur costs of internal failure. These are the actual costs per bug.
- Cost of external failure: The support, testing, development, and other costs that incur because no 100% bug-free software was delivered. For example, costs for technical support, help desk organization, and sustaining engineering teams are costs of external failure. These costs are much higher. The costs are at least as high as the internal failure costs, but there are additional costs for external communication, containment, and external support.
Let’s define the Average Cost of Test Bug with the knowledge from above:
ACTB = (cost of detection + cost of internal failure) / (test bugs) [$/Bug]
There is also the definition of the Average Cost of a Production Bug:
ACPB = (cost of external failure) / (production bugs) [$/Bug]
With both definitions from above the calculating of a Test Return On Investment can be defined:
TestROI = ( ( ACPB – ACTB) x test bugs) / (cost of detection) [1]
Example Test ROI Calculation
Let us assume, we have 5 developers in a test team which cost about 50k$ per month (salary and all other costs). Let us further assume, they find an average of 50 bugs per month and a customer finds 5 bugs a month. The production bug cost is assumed with 5k$ (5 people for a week for customer support, bug fixing and negotiations) and a testing bug cost of 1k$ (1 developer for a week). We have the following numbers:
ACTB = (50 k$/month + 50 bug/month * 1k$/bug) / (50 bug/month) = (100 k$/month) / (50 bug/month) = 2k$/bug
ACPT = (10 bug/month * 5 k$/bug) / (10 bug/month) = 5 k$/bug (what we obviously defined)
TestROI = ((5 k$/bug – 2k$/bug) * 50 bug/month) / (50 k$/month) = 3
This simple example shows, that testing might be economically meaningful…
Automated and Manual Regression Tests
Some simple formulas for automated testing. It is sometimes difficult to communicate the positive impact of automated testing, because the initial costs for hardware, operational costs and the costs for development of automated tests are significant. But, the next formulas give an impression for the benefit.
Let us define the Regression Test Automation ratio:
RTA = (automated regression tests) / (manual tests + automated regression tests)
The Regression Risk Coverage is shown here to have the complete set of formulas:
RRC = (regression risk covered) / (regression risks identified)
The RRC is used as a measurement for the confidence on the automated tests. The higher the RRC, the better the confidence not to deliver critical bugs.
To calculate the benefit of automated test, the Acceleration of Regression Testing can be used:
ART = (manual regression test duration – automated regression test duration) / (manual regression test duration)
Automated tests bring benefits twofold:
- Development and test engineers are relieved from repetitive work. Engineers in most cases are not very good in this. Engineers are better placed inside of development projects and in failure analysis which brings more economical benefits. Every hour saved for manual testing can be better spend in functionality development or quality improvement.
- Automated tests can be run anytime, even during night. Continuous Integration (CI) and Continuous Delivery (CD) approaches use this by automated triggering of tests. Tests are run for example after each code change in the Source Code Management tool. Bugs can be found and fixed very quickly.
The costs can be lower, too. If the testing environment costs 1k$/month, it is about 1/5 of a developer, but it runs 24/7 and executes much more tests per time unit than any developer can do. It is a real bargain.
Summary
Even it is only a short post on some defect cost calculations, it shows that testing is meaningful economically. The costs for testing like developers developing automated tests, dedicated testers doing manual tests, hardware, QA staff managing testing and assuring the quality compliance, and so forth seem very high, but taken into account that the costs for defects increase significantly the later they are found in the product life cycle, testing brings a huge benefit.