Metrics - Test Preparation & Planning

Following the defect metrics post I came across my list metrics that I find useful in projects. The metrics focus on 3 areas of the test activities: Test Planning, Test Execution and Test Results – This post will be about Test Preparation (& planning).

For Test Preparation & planning I’ll invite you considering the following:

·         Test Plan Documentation

·         Test Preparation

·         Test Coverage

·         Test Estimates

I have included some recommendations to targets to achieve, risks you face by not monitoring this as inspiration. These are but a few things that you can measure against, but experience is that simple metrics that has a very specific purpose in life will get you far.

Happy testing!

/Nicolai

 

Test Plan Documentation     

Measured as: 

·         Does a test plan document exist?

·         Is the test plan formally approved/agreed with the stakeholders?                         

Purpose / Objective:

·         This metric measure if test plans are documented and discussed and approved with the stakeholders.         

Recommended targets:

·         Test plan is created early.

·         Test plan is formally approved by stakeholders.    

Measurement Frequency

·         One off measurement as part of project start up.

Risks if not monitored

·         Unplanned and unmanageable test.

·         Expectations to test not aligned with stakeholders.

·         Resources & estimates for test not outlined early.

Metric: Test Preparation

Measured as: 

·         Number of test cases per test level.

Purpose / Objective:

·         This metric measure if test plans are documented and discussed and approved with the stakeholders.         

Recommended targets:

·         All test levels must be covered by test cases.

·         Number of test cases should reflect complexity / risk.

·         Test levels with broad scope will often have many cases.

Measurement Frequency

·         Prior to test level readiness review and on ad hoc.

Risks if not monitored

·         Functional areas / requirements not covered by test.

·         No early test execution = cost escalation of defects.

Metric: Test Coverage

Measured as: 

·         Requirements not covered by test cases.                

Purpose / Objective:

·         This metric aims to uncover requirements that are not covered by test, in order to flag risk and assist test team in getting appropriate level of coverage in the test.

Recommended targets:

·         All requirements should be covered by test cases.

·         Traceability between test cases and underlying requirements must be possible to do prober coverage analysis.            

Measurement Frequency

·         Prior to test level readiness review and on ad hoc.

Risks if not monitored

·         Unknown quality in delivery.

·         Untested functionality released to production.

Metric: Test Estimates

Measured as: 

·         Work breakdown of test activities in test plan for all test levels  

Purpose / Objective:

·         This metric aims to verify prober level of work breakdown and to ensure that estimates are detailed enough to describe the activities.

Recommended targets:

·         All test levels must have estimates that quantify the test effort needed for each test level.

·         Test estimates must be detailed enough and broken down to less than one man-week of effort.

Measurement Frequency

·         Prior to negotiation of test budget and planning activities

Risks if not monitored

·         Inaccurate / unrealistic estimates for test activities.

·         Inaccurate planning and inability to deliver against major milestones.

Monitoring your defects

Problem: Unmonitored defects will turn into technical debt

You have found a bucket load of defects during testing, so far so good, but they need constant care in order to avoid them ending up as technical debt.

Solution: Defect trend monitoring and tracking

You need to arm your organization with a few good metrics that will give you an overview of state of affairs and allow you to see trends in your defect database. There are plenty of metrics to choose from, especially if you are using a modern defect tracker, but in my experience less is more in this context.

Consider the following metrics:

·         Defect arrival rate: Graph that shows progress over time.

·         Severity and priority spread: Pie that shows what severity and priority open defects have

·         Defect turnaround time & Defect ageing: Time from discovery until closure & time since last update of the defect in your database.

Each of the metrics will tell you something about the state of affairs in the item under test, and here is what you should look for:

Defect arrival rate:

I usually just look at 3 lines, open, closed and total defects in order to get the following information:

Total defects:

·         Are we testing? – Flatline means no.

·         Is quality improving? – Slowdown in arrival either indicates that chances of meeting release goals increases. If you do not see a slowdown in arrival in the weeks prior to shipping the software you should expect a large volume of defects found in production after golive.

Open/closed defects:

·         Closure rate - Is the gab between open and closed defects shrinking or increasing? Extrapolating the trend will tell you if you can expect lots of known bugs to ship with product or not. Something that comes in handy when doing expectation management with the customer receiving the product.

Example:

·         It seems that testers was not testing much in weeks 6-8, as total defects are stable on 100

·         There is a slowdown in defects from week 9 to 10, but this might just be coincidence, so do not draw conclusions on trends until you see numbers from week 11.

·         Defect closure rate seems constant, and open defects are dropping. This is healthy if the product ships soon, and we assume that the test is nearing completion.

Severity & Priority Spread:

Look for high volumes of high priority or severity defects. In case you find yourself in a situation where you have too many high ones then you need to stop and do a bug triage, or you risk loosing the ability to use severity & priority to steer your efforts. This is detailed in one of my previous posts found here:

http://testing4tw.blogspot.dk/2013/09/when-defect-priority-looses-its-meaning.html

Defect turnaround time & ageing:

These two metrics tells you something about your organization’s ability to process defects found. What you should look for are

·         Do we address our defects in a timely manner? Looking at time since last update for defects with high severity and/or priority is interesting when talking to customers and other stakeholders in a project. Furthermore this will help you determine if there is a problem getting those critical ones fixed in due time.

·         Knowing your defect fix rate or capacity will make it much easier when estimating for future planning, meaning that you get some insight for the future.

Happy defect monitoring!

/Nicolai

Baselining test data in your test environment

This post is inspired by project experience. Bitter project experience.

The project was delayed. Classical V-model approach with a couple of rounds of system test and system integration test at the end of the plan. Development turned out to be even more problematic for a number of reasons mostly attributed to the integration betweeen the systems of an internal and an external supplier.

The test team used the waiting time for preparation so when the day came and the environment was smoke tested all pressure lay on the team - and they were ready as ever. Since the additional preparation effort paid off testing managed to stay within the plan for the first round of testing. Thumbs up.

Bugs were fixed, testing started once again and then some more bugs were discovered. An additional round of testing was agreed and planned and just before that was about to start - disaster was discovered.

The configuration of the test environment was flawed. Big time. All the integrations were set up correctly, technical users and other trivial stuff worked but the data and configuration of static data was wrong. All test results were invalid. All bugs were false positives. Everybody, including the project manager, had been so eager to get testing started that nobody had bothered questioning the baseline of the test data configuration. And the initial smoke test had proven that the environment was fully operational for testing - but not that data was correct.

Two weeks wasted. All energy sucked out of the project.

Lesson learned?

Prepare and describe the test data baselining test activities! Especially if test data is crucial to your test.

• Identify crucial test data as early as possible
• Understand how to validate that data in the environment before testing starts
• Perform a review to get a second opinion
• Plan a smoke test that focuses on proving that data is present in the desired state in the test environment
• Execute the smoke test
• Analyse and understand test result.

The bullets mentioned above might require a few extra hours of planning and preparation - and a very limited test effort. But they will save your project from at least one major risk. The risk of having a perfectly planned test executed - with no value added to the project.

Proving that test data is correct is one of the most tricky tasks for a test team. But a little effort can determine whether it feels right or wrong to go ahead with the test.

What's your test data strategy??

Through my years as a test manager I've seen many test strategies and plans. Most of them included a section on "test data". Often a short one mentioning the need for test data and the types of data listed in a table - that's it.

For some projects it's enough but more often I've come to realise that data is the test  - or data is driving the test. So there is a need to dig deeper and to be a little more focused on what counts as "test data".

ISTQB defines test data as: Data that exists (for example, in a database) before a test is executed, and that
affects or is affected by the component or system under test.

So far so good. There is also a definition of test data management:

The process of analyzing test data requirements, designing test data structures, creating and maintaining test data.

That is a lot more interesting because it touches upon a whole set of prerequisites for handling data - and ensuring that data is available in a controlled way. Even the most elaborate test cases, test analysis documents, test environments and skilled test team is worthless without data. And ever so often you see panicking people frantically trying to create data on the fly to pass that last important test.

So starting thinking about a structured approach to test data asking questions like:

• Test data landscape - what's the total set of test data
• Ownership of test data - system or organisation that "owns" data
• Re-useability of test data - can data be handled in a smart wat
• "Consumeable" test data - test data that can only be used once
• Manipulation of test data - how to create data that can be used in a certain test context.
• Primary types of test data - identify test data dependencies where one type of data is directly related to others.

These are probably some of the questions that will bring forward a lot of other relevant issues and challenges in a test environment where there are a number of integrated systems included in test. Once there are integrations outside your own physical organisation you must have a strategy - or trial and error will prove to be very expensive.

User reference group studies...

A colleague of mine showed me this video, and aside being funny (and a little disgusting) it has some points that I think is worth noticing – watch the clip before reading the rest, to prevent my comments ruining your laugh.

Remember to turn the sound on, the comments are priceless.

https://www.youtube.com/watch?v=m-wQhSuf6mM

What we see is what could be perceived as a user reference group. They are not likely to be the typical consumer of Surströmming, but they serve as excellent reference for users without knowledge of the product would experience.

Here is what I noticed:

• There is a high enthusiasm, and expectation to this product. The group is clearly under influence of expectations build prior to the video. Be aware that you do not accidentally influence the group when setting up / explaining the concept to them.
• The group is prepared, they have brought the tools for the task. This is very important, as the users’ opinion and reaction will be affected by lack of preparation, so make sure that the usability study is well prepared.
• The opening of the product is catastrophic, this is definitely something that would raise a bug in the usability department. A label reading “handle and open with care” seems to be missing, or has escaped the attention of the users.
• If product is so bad that it makes the people leave the area, you might want to rethink your design/implementation.
• The user reference group invites in a new member – The dog. This is just a bonus for the test, make sure to capture information you get for free.
• The user reference group shares associations about the product. Statements like “This reminds me of…” are very powerful, as it tells something about the mood and feelings that the user is experiencing. Capture them and ask questions to the story after the session.

Happy testing!
/Nicolai

Celebrate your success!

We celebrated the completion of test procedures for usecases related to Release 1 today, and what is better than celebrating it with some ”use cake”.

 

I would like to take the opportunity to wish you the best for the summer and remind you that you should celebrate your successes and not your failures.

Happy testing!

/Nicolai

The Cost of Finding Defects

Problem: Calculating the cost of finding a defect

One of my peers asked me how I would calculate the cost of finding a bug the other day. Reason for the quastion was that management was doing a RoI calculation on the test effort in the project portfolio, and would like to know how much effort (read: money) they would be spending to find a single bug.

Solution: Guesstimating based on a few metrics

In nature knowing the price-tag for finding a defect is interesting, and my answer to the question was something like this:

Gather some metrics from your test management tool, test plans, timelogging system or similar.

Something like total number of test cases (TC) and defects fount, hours spend on TC preparation and TC execution

Apply simple math like:

((hours spend on TC preparation + hours spend on TC execution)/ total test cases)* Defect to test case ratio (Defects / TC)

For example:

((442 + 159)/102)*(98/102) = 5,66 man hours / defect.

That excludes all the cost like test environment cost, training of testers, time spend on defect management etc.

Happy guesstimating!

/Nicolai