Monitoring your defects

Problem: Unmonitored defects will turn into technical debt

You have found a bucket load of defects during testing, so far so good, but they need constant care in order to avoid them ending up as technical debt.

Solution: Defect trend monitoring and tracking

You need to arm your organization with a few good metrics that will give you an overview of state of affairs and allow you to see trends in your defect database. There are plenty of metrics to choose from, especially if you are using a modern defect tracker, but in my experience less is more in this context.

Consider the following metrics:

·         Defect arrival rate: Graph that shows progress over time.

·         Severity and priority spread: Pie that shows what severity and priority open defects have

·         Defect turnaround time & Defect ageing: Time from discovery until closure & time since last update of the defect in your database.

Each of the metrics will tell you something about the state of affairs in the item under test, and here is what you should look for:

Defect arrival rate:

I usually just look at 3 lines, open, closed and total defects in order to get the following information:

Total defects:

·         Are we testing? – Flatline means no.

·         Is quality improving? – Slowdown in arrival either indicates that chances of meeting release goals increases. If you do not see a slowdown in arrival in the weeks prior to shipping the software you should expect a large volume of defects found in production after golive.

Open/closed defects:

·         Closure rate - Is the gab between open and closed defects shrinking or increasing? Extrapolating the trend will tell you if you can expect lots of known bugs to ship with product or not. Something that comes in handy when doing expectation management with the customer receiving the product.

Example:

·         It seems that testers was not testing much in weeks 6-8, as total defects are stable on 100

·         There is a slowdown in defects from week 9 to 10, but this might just be coincidence, so do not draw conclusions on trends until you see numbers from week 11.

·         Defect closure rate seems constant, and open defects are dropping. This is healthy if the product ships soon, and we assume that the test is nearing completion.

Severity & Priority Spread:

Look for high volumes of high priority or severity defects. In case you find yourself in a situation where you have too many high ones then you need to stop and do a bug triage, or you risk loosing the ability to use severity & priority to steer your efforts. This is detailed in one of my previous posts found here:

http://testing4tw.blogspot.dk/2013/09/when-defect-priority-looses-its-meaning.html

Defect turnaround time & ageing:

These two metrics tells you something about your organization’s ability to process defects found. What you should look for are

·         Do we address our defects in a timely manner? Looking at time since last update for defects with high severity and/or priority is interesting when talking to customers and other stakeholders in a project. Furthermore this will help you determine if there is a problem getting those critical ones fixed in due time.

·         Knowing your defect fix rate or capacity will make it much easier when estimating for future planning, meaning that you get some insight for the future.

Happy defect monitoring!

/Nicolai