Metrics - Test Results

Next up in the metrics posts are the metrics for Test Results, where we will look at metrics for defect Accuracy, Test Efficiency and Technical Debt.

I have included some recommendations to targets to achieve, risks you face by not monitoring this as inspiration.

Happy testing!

/Nicolai

  

Metric: Defect Accuracy

Measured as: 

·         Defects that are or has been in status Reopen, rejected or duplicate.

Purpose / Objective:

·         This metric aims to pinpoint waste in the defect management life cycle. Defects that are not accurate will waste resources in the defect management lifecycle.

Recommended targets:

·         More than 15% inaccurate defects require actions from defect manager to ensure that efforts are not wasted.

Measurement Frequency:

·         Weekly while test is running and prior to test readiness reviews

Risks if not monitored:

·         Waste of resources.

·         Poor defect management.

Metric: Test Efficiency

Measured as: 

·         Defect Detection Ratio (DDT) vs. previous test phase - Number of defects found 14 days after the end of a test divided by the number of defects found in the test level under assessment.

Purpose / Objective:

·         This metric aims to give an indication on how efficiently the recently conducted test was.

·         Low DDT ratios indicate that project is subject to cost escalation due to defects being found too late in the process.

Recommended targets:

·         80% DDT for the later test levels as minimum, to reduce cost escalation. 85%+ for UAT test level.

Measurement Frequency:

·         Should be executed 14 days after each test level & 14 days after go-live to prod.

Risks if not monitored:

·         Inability to see value for money for each test phase.

·         Inability to prioritize test effort for new releases / iterations.

Metric: Technical debt

Measured as: 

·         Age of defects that are not closed.

Purpose / Objective:

·         This metric serves as input to defect prioritization and management of the defects found as part of any test.

Recommended targets:

·         No defects older than 6 months.

·         No high severity defects older than 1 month.

Measurement Frequency:

·         Weekly or in agreement with stakeholders

Risks if not monitored:

·         Unknown technical debt.

·         Problems scoping new releases / iterations.

Metrics - Test Progress & Control

Next up in the metrics posts are the metrics for Test Progress & Control, where we will look at metrics for Test Progress, Product Quality and Defect Resolution

I have included some recommendations to targets to achieve, risks you face by not monitoring this as inspiration.

Happy testing!

/Nicolai

Metric: Test Progress

Measured as: 

·         S-curve planned test cases vs. executed test cases

Purpose / Objective:

·         This metric aims to ensure that the test is running as planned, and serves as a tool to see trends of delays for risk mitigation and replanning.

Recommended targets:

·         Test execution must be monitored for each test level.

Measurement Frequency:

·         Weekly while test runs or in agreement with stakeholders

Risks if not monitored:

·         Unmanaged test.

·         Inability to proactively meet delays.

·         Unknown progress.

Metric: Product Quality

Measured as: 

·         Number of failed test cases vs. number of run test cases

Purpose / Objective:

·         This metric gives stakeholders an indication on the product quality, showing the amount of test failure for the object under test.

Recommended targets:

·         Trends of high failure should be followed by a test readiness review ensure that test resources are not wasted testing immature deliveries.

Measurement Frequency:

·         Weekly while test is running and prior to test readiness reviews

Risks if not monitored:

·         Unknown product quality.

·         Inability to mitigate poor quality early.

Metric: Defect Resolution

Measured as: 

·         Defects that are open and closed and the severity of these defects.

·         Measurement should be as progress over time to see trend and a snapshot to get overview of severity of defects.

Purpose / Objective:

·         This metric gives an indication on the amount of rework that is outstanding and shows trends of test completeness.

Recommended targets:

·         Critical defects must be closed prior to release of a product as stated in Nordea’s Corporate test policy.

Measurement Frequency:

·         Weekly while test is running and as part of test evaluation after each test level.

Risks if not monitored:

·         Poor or missing defect management.

·         Unknown size of the work backlog related to defects.

·         Inability to see defect trends and test completeness.

·         Unknown defect severity = inability to prioritize defects for closure.

Metrics - Test Preparation & Planning

Following the defect metrics post I came across my list metrics that I find useful in projects. The metrics focus on 3 areas of the test activities: Test Planning, Test Execution and Test Results – This post will be about Test Preparation (& planning).

For Test Preparation & planning I’ll invite you considering the following:

·         Test Plan Documentation

·         Test Preparation

·         Test Coverage

·         Test Estimates

I have included some recommendations to targets to achieve, risks you face by not monitoring this as inspiration. These are but a few things that you can measure against, but experience is that simple metrics that has a very specific purpose in life will get you far.

Happy testing!

/Nicolai

 

Test Plan Documentation     

Measured as: 

·         Does a test plan document exist?

·         Is the test plan formally approved/agreed with the stakeholders?                         

Purpose / Objective:

·         This metric measure if test plans are documented and discussed and approved with the stakeholders.         

Recommended targets:

·         Test plan is created early.

·         Test plan is formally approved by stakeholders.    

Measurement Frequency

·         One off measurement as part of project start up.

Risks if not monitored

·         Unplanned and unmanageable test.

·         Expectations to test not aligned with stakeholders.

·         Resources & estimates for test not outlined early.

Metric: Test Preparation

Measured as: 

·         Number of test cases per test level.

Purpose / Objective:

·         This metric measure if test plans are documented and discussed and approved with the stakeholders.         

Recommended targets:

·         All test levels must be covered by test cases.

·         Number of test cases should reflect complexity / risk.

·         Test levels with broad scope will often have many cases.

Measurement Frequency

·         Prior to test level readiness review and on ad hoc.

Risks if not monitored

·         Functional areas / requirements not covered by test.

·         No early test execution = cost escalation of defects.

Metric: Test Coverage

Measured as: 

·         Requirements not covered by test cases.                

Purpose / Objective:

·         This metric aims to uncover requirements that are not covered by test, in order to flag risk and assist test team in getting appropriate level of coverage in the test.

Recommended targets:

·         All requirements should be covered by test cases.

·         Traceability between test cases and underlying requirements must be possible to do prober coverage analysis.            

Measurement Frequency

·         Prior to test level readiness review and on ad hoc.

Risks if not monitored

·         Unknown quality in delivery.

·         Untested functionality released to production.

Metric: Test Estimates

Measured as: 

·         Work breakdown of test activities in test plan for all test levels  

Purpose / Objective:

·         This metric aims to verify prober level of work breakdown and to ensure that estimates are detailed enough to describe the activities.

Recommended targets:

·         All test levels must have estimates that quantify the test effort needed for each test level.

·         Test estimates must be detailed enough and broken down to less than one man-week of effort.

Measurement Frequency

·         Prior to negotiation of test budget and planning activities

Risks if not monitored

·         Inaccurate / unrealistic estimates for test activities.

·         Inaccurate planning and inability to deliver against major milestones.

Monitoring your defects

Problem: Unmonitored defects will turn into technical debt

You have found a bucket load of defects during testing, so far so good, but they need constant care in order to avoid them ending up as technical debt.

Solution: Defect trend monitoring and tracking

You need to arm your organization with a few good metrics that will give you an overview of state of affairs and allow you to see trends in your defect database. There are plenty of metrics to choose from, especially if you are using a modern defect tracker, but in my experience less is more in this context.

Consider the following metrics:

·         Defect arrival rate: Graph that shows progress over time.

·         Severity and priority spread: Pie that shows what severity and priority open defects have

·         Defect turnaround time & Defect ageing: Time from discovery until closure & time since last update of the defect in your database.

Each of the metrics will tell you something about the state of affairs in the item under test, and here is what you should look for:

Defect arrival rate:

I usually just look at 3 lines, open, closed and total defects in order to get the following information:

Total defects:

·         Are we testing? – Flatline means no.

·         Is quality improving? – Slowdown in arrival either indicates that chances of meeting release goals increases. If you do not see a slowdown in arrival in the weeks prior to shipping the software you should expect a large volume of defects found in production after golive.

Open/closed defects:

·         Closure rate - Is the gab between open and closed defects shrinking or increasing? Extrapolating the trend will tell you if you can expect lots of known bugs to ship with product or not. Something that comes in handy when doing expectation management with the customer receiving the product.

Example:

·         It seems that testers was not testing much in weeks 6-8, as total defects are stable on 100

·         There is a slowdown in defects from week 9 to 10, but this might just be coincidence, so do not draw conclusions on trends until you see numbers from week 11.

·         Defect closure rate seems constant, and open defects are dropping. This is healthy if the product ships soon, and we assume that the test is nearing completion.

Severity & Priority Spread:

Look for high volumes of high priority or severity defects. In case you find yourself in a situation where you have too many high ones then you need to stop and do a bug triage, or you risk loosing the ability to use severity & priority to steer your efforts. This is detailed in one of my previous posts found here:

http://testing4tw.blogspot.dk/2013/09/when-defect-priority-looses-its-meaning.html

Defect turnaround time & ageing:

These two metrics tells you something about your organization’s ability to process defects found. What you should look for are

·         Do we address our defects in a timely manner? Looking at time since last update for defects with high severity and/or priority is interesting when talking to customers and other stakeholders in a project. Furthermore this will help you determine if there is a problem getting those critical ones fixed in due time.

·         Knowing your defect fix rate or capacity will make it much easier when estimating for future planning, meaning that you get some insight for the future.

Happy defect monitoring!

/Nicolai

Baselining test data in your test environment

This post is inspired by project experience. Bitter project experience.

The project was delayed. Classical V-model approach with a couple of rounds of system test and system integration test at the end of the plan. Development turned out to be even more problematic for a number of reasons mostly attributed to the integration betweeen the systems of an internal and an external supplier.

The test team used the waiting time for preparation so when the day came and the environment was smoke tested all pressure lay on the team - and they were ready as ever. Since the additional preparation effort paid off testing managed to stay within the plan for the first round of testing. Thumbs up.

Bugs were fixed, testing started once again and then some more bugs were discovered. An additional round of testing was agreed and planned and just before that was about to start - disaster was discovered.

The configuration of the test environment was flawed. Big time. All the integrations were set up correctly, technical users and other trivial stuff worked but the data and configuration of static data was wrong. All test results were invalid. All bugs were false positives. Everybody, including the project manager, had been so eager to get testing started that nobody had bothered questioning the baseline of the test data configuration. And the initial smoke test had proven that the environment was fully operational for testing - but not that data was correct.

Two weeks wasted. All energy sucked out of the project.

Lesson learned?

Prepare and describe the test data baselining test activities! Especially if test data is crucial to your test.

• Identify crucial test data as early as possible
• Understand how to validate that data in the environment before testing starts
• Perform a review to get a second opinion
• Plan a smoke test that focuses on proving that data is present in the desired state in the test environment
• Execute the smoke test
• Analyse and understand test result.

The bullets mentioned above might require a few extra hours of planning and preparation - and a very limited test effort. But they will save your project from at least one major risk. The risk of having a perfectly planned test executed - with no value added to the project.

Proving that test data is correct is one of the most tricky tasks for a test team. But a little effort can determine whether it feels right or wrong to go ahead with the test.