Using defect trends for estimation

Problem: Predicting amount of defects in the week to come is difficult.

Being able to guess the amount of defects found in the week(s) to come is very useful for ressource estimation, risk evaluation and other test managment related activities. Furthermore this require a little crystal ball technology, as it requires the test manager to make a qualified quess, based on assumptions related to the test items and resources.

Solution: Use your defects trends as guide for the estimation.

Being able to make qualified guesswork require that you use your metrics as foundation for seeing trends. The low-tech approach that I use is to extrapolate the trend and see where total will be end of next week and the week after. The cool thing about low tech solutions is that they are easily applied and communicated,

Consider this example, where test has been running for 2 weeks and question from development team is how much effort shall we reserve for bug-fixing. This question is in fact: “How many defects should we expect raised next week?”

Using the low tech approach I just draw a line that follows the trend from the previous weeks of testing. That gives an indication on where the defect count will end if all test conditions are the same.

The following factors can should then be taken into account before guessing the total defects next week:

• Is the test resources and cases for the week to come approximately the same as the previous weeks?

• Will items under test change so the test team gets new or complex deliveries for test?

• Is there a lot of retest in the week to come? Will this take resources away from the test?

• Do you have planned maintenance in the test environment?

These assumptions are then used to adjust the slope of the trendline that you base your guesstimate. In above example the trendline is set higher than average, as the test items in week 3 are highly complex and 1st time under test. This lands the defect total on 85+, or additional 35 defects in week 3 (calculated as week 3 total minus week 2 total).

You can also estimate severity for the defects to come in the following way: Combine your trend graph with your defects severity trends then you can add severity as a flavor to the estimate. In above example project we have the following severity distribution after week 2: 7% Critical 13% Major 25% Medium 37% Minor and 18% Cosmetic. This leaves week 3’s 35 estimated defects with the following spread:  2,5 Critical 4,5 Major 8,75 Medium 13 Minor and 6,25 Cosmetic

There is no rocket science in the equation, but only simple logic based on extrapolation of trends that is easily communicated to the recipient. Putting the graph with trend line and your assumptions in a mail is a short route to setting expectations. There are also other situations where this approach will come in handy, when communicating general quality trend in a delivery or when talking to customers or steering committees while doing expectation management.

Happy testing & Enjoy your weekend!

/Nicolai