When defect priority looses its meaning... STOP!

Problem: All defects are critical just before launch – defect priority can easily loose it’s value & meaning

Clock is ticking and a release is approaching. Development only have a few days/hours before code-freeze and only the most urgent bugs can be addressed. This is where defect priority can loose its meaning, as people see all bugs as critical, hoping for them to make the cut.

Solution: STOP! Check your defect metrics and call for a bug triage

Most extreme example I have seen was a large program where the % of priority 1 & 2 defects went from 17% to 92% in a week. This is what happened:

Program management announced to all stakeholders that deadline was tight, and that time would no longer pay attention to defects of severity 3 or lower. In the two days that followed this announcement, we could see a pattern forming. More than half of the existing defects had priority changed, majority got priority 1 or 2. 90+% of new defects was opened with priority 2+.

This story holds two lessons:

• Do not tell stakeholders that you will be ignoring their defects – They will not take it kindly.
• Make sure to check your defect metrics from time to time and do a bug triage when needed.

Since this I have always kept a pie-chart of defect priority (and severity) at hand in my projects – This is easy, as you get it for free in most defects tracking tools. Checking it from time to time, will offer a sanity check of priority spread, before doing a bug triage exercise.

In the example above priority got confused with severity, everyone got all jumpy about when, and if things got fixed. This meant that the fist bug triage meeting was a battle of will – Who was prepared to give ground and sacrifice some bugs to lower priorities? The meeting did not change any priorities, and we simply agreed that all stakeholders (business areas) had to nominate 3 bugs that they needed and 5 they wanted, and that set the new standards for priorities – Not very pretty but it did the job, giving development priorities for completing bug fixing in the time left.

For more information on bug triage have a look at this: http://www.softwaretestingtimes.com/2010/04/bug-triage-meeting-process.

Have a nice weekend & Happy testing!

/Nicolai

Testing ROI - what's your story??

Problem:

There are numerous ROI calculations on various forms of testing. Be it manual or automated testing, in waterfall or agile contexts someone has crunched the numbers for "testing as a whole" or for a specific industry.
It very often leads to some rather academic figures.

And very often to a setup where a lot of assumptions or "model projects" have to form the foundation for the calculations - and then another set of "best practices" on top of this with estimates on how many minutes it takes to do a test script, review and execute it, how many defects/test script. And then you have to put in the human factor - multiply by 1.07 and the ROI is....lost.


Solution:

Find a good "product" story for your project or for your QA department in your company. You will most likely not be able to do flashy ROI on top of the story. Or only partially.
I've been with a company that was very focused on software- and hardware quality, yet what we needed was not ROI to remind us that test was important. We all shared the same story which was told when you joined the company - two failed product launches and we are out of business.
And QA didn't want to contribute to that in any way. At all. So we tested because we had to do test - because we did not want to fail. And that did not change for years. ROI calculations were updated during that time, better tools to support testing were introduced, the test deparment matured in terms of processeses and participants skills and experience but at the end of the day our story was still the selling point. And it could be shared outside the QA department with other deparments.

I'm not saying that ROI's are useless. Some are obviously so easy to pick apart that they are. On the other hand good ROI's are great input for understanding how much effort and reuse it takes before automation is feasible. And it is also good for telling the story about how many parameters that are impacting testing. In short the ROI process is good for getting discussions and understandings aligned internally. If there is a factor of 20 between two estimates or factors in an ROI where is the consensus and why?

If you are put with the task of doing a ROI to justify a test department in your company you should consider something to be fundamentally wrong. Test departments are needed whenever there is software or hardware development. They can be outsourced with different disasters waiting to happen. Unless you also ROI "communication challenges". And if you do that please share your thoughts on this blog.
But we would much rather hear your product story.

 

Test automation return of investment

Problem: Calculating & using Return of Investment (RoI) in test automation implementation

Engaging in discussions on RoI when implementing test automation is a tricky business. Discussions is often derailed either by incorrect numbers, a strange formula being used or stakeholders who for political or personal reasons wants to influence the decision.

Solution: Expectation management and a simple RoI calculation to steer the discussion

RoI calculations are needed for doing the business case detailing the investment that test automation is. There is really no way around this, but I would recommend that you start your test automation discussion with expectation management.

Expectation management is needed, as there will be many different opinions on what can, and what cannot be done using test automation. Especially the falsely expected benefits are dangerous, as they will pull discussion in the wrong direction, and set the project up for failure. When looking for falsely expected benefits and other mistakes I suggest that you start the discussion with stakeholders listing the tangible and intangible benefits they see. This will serve as input for the business case and give you an idea of the realism in the expectations of the stakeholders.

Test automation is an investment that takes a lot of time to get RoI from. Especially the arguments about savings and cost reduction should be examined closely and that is where a RoI calucation will come in handy. Some years ago I got a copy of Dorothy Graham’s test automation calculator, a nice Excel sheet that allows you to make a simple RoI. You can find it here, along with some notes on the subject on Dorothy's Blog.

In my experience RoI calculations can never stand alone – The numbers in the calculation needs to be backed by the experience you get from doing the actual test automation. When introducing test automation to a project, I suggest that you do a short proof of concept to ensure that the assumptions are right and that estimates are reasonable.

In short:

• Engage stakeholders in a discussion on expectations and manage those
• Do a RoI based on a simple calculation – Remember to mention the intangible benefits
• Make a proof of concept, where you check your numbers and assumptions

Want to know more? Check out Douglas Hoffman’s excellent paper on ’Cost Benefits Analysis of Test Automation’ found here: http://www.softwarequalitymethods.com/papers/star99%20model%20paper.pdf


Happy testing!

/Nicolai
 

Test policy considerations

Problem:

"Test" is a holistic expression used randomly across the organisation until 2 (weeks/days/hours) before go-live.

You've worked with test for more than six months, you know the above is true. Some exceptions exist but they are exceptions.

Solution:

Write an ultra short test policy and if organisational acceptance fails, at least you know that things will never change.

A fellow test blogger has done an interesting blog entry about test policies (in Danish). So based on her headlines we tried to come up with the shortest possible test policy that can be adapted to almost all organisations.

Why not a 20 page document? Because they are never read, never updated and never detailed enough to deal with the fact that IT-development is a one giant moving target, no two projects are alike and change happens fast.

So instead of spending months and weeks on that unpolished jewel, spend 1 hour. Preferably with fellow test professionals and see how short you are able to fill in 9 sections below - or if they are relevant for you at all.

Test policy for you

1. Definition of- and purpose of test in the organisation. This organisation recognises the importance of structured test, with the aim of providing valuable and tangible information to relevant parts of the organisation and the senior management group.
2. Test is organised according to demand since it is a support activity to 1) projects of whatever size, 2) maintenance and 3) IT-operations. Test is a distinct discipline and thus also has its own distinct manager (insert name).
3. Career tracks and competence development of the testers is subject to demand from the organisation. Competence planning will be carried out at least two times a year or at the end of each (project) assignment.
4. The overall test process that must be followed is embedded in the 1) project model and the 2) development model that are implemented in the company. No stand-alone test models are accepted. The implementation of test activities and related gates are evaluated and adjusted when the models are up for revision.
5. Standards that must be followed. Since test is a professional discipline we follow the vocabulary of ISTQB as a standard. In the case we are working within a regulated industry we follow (insert name) standard(s).
6. Other relevant internal and external policies that must be followed - refer to (4)
7. Measurement of test value creation is based on the following three criteria:
• Input from test in terms of blocking defects (number of accepted, requirements traceable bugs that have required a fix)
• Input from test in terms of implemented process improvement suggestions (time to market improvements).
• Assistance with defect prevention activities (number of accepted bugs found during analysis and inspection of requirements and other static test environments)
8. Archiving and reuse of test results are determined at the end of each task or project.  A peer review will be conducted to determine which test artefacts are worth saving for re-use, and which existing test artefacts must be discarded.
9. Ethics for the testers:
• Bugs must be documented and made visible to the organisation
• You own your own bugs until handover is accepted by the receiver
• All testers are expected to speak up when they come across something "fishy" i.e. improper solutions, processes, implementations or the like.

Cutting corners has a price…

Problem: Risk impact discussions can be problematic

Someone posted this picture at Linkedin, and that made me think of the iron triangle. While working with quality you often have to talk about the cost of quality. Quality is fluffy, and hard to explain, especially when talking quality in large projects. As a tester you have valuable input for the risk logs and risk meetings - Make sure that you get the point across to the receiver.


Solution: Use the iron triangle to illustrate the consequences of a risk to your peers

I often use the iron triangle when talking impact of risks and issues. The reason for this is that it is simple and speaks a language that most project managers will understand.

This is how I do it:

Every time I identify something that threatens quality, I think about the triangle, and ask the question; where will this hurt the most? Cost, Scope or Schedule? I really like this angle on risk analysis, as it gives much to the risk index from traditional risk management.

The thing this approach adds, is easier prioritization amongst risks based on the overall project goals, as most project managers will be able to say which of the corners of the triangle he can afford to cut. Furthermore, it links the risk impact score directly to something that project members can relate to.

Give it a try – I bet it will raise awareness about risk impact in your project. At least it will give you a nice discussion on what corner your project will cut off the triangle when things gets though.

One last thing: Cutting corners always costs quality...

Have a nice day & Happy testing!

/Nicolai

Get to know your defects

How many times have you been in a project where the standard defect report has been published - and nobody cared?

The numbers are right, the defects are grouped according to severity, system area, functional area or whatever makes sense in the project. If you are in a well-managed project you'll also be presented to an overview showing changes over time. The report is filed with graphs for easy presentation of the numbers. Nobody is awake now. Why?

 Well, the "why" word is exactly the reason nobody cares about standard reports because they do hardly say a thing about why the defects are there.

Its' comparable to a train wreck. It has happened. The dust has settled. What remains now is twisted metal and a lot of questions with few obvious answers.

In well-driven projects there's room for the most fabulous defect activity - root cause analysis. The kind of work that really pays off both in the short run and in the long run.

The most fantastic book I've read for a while is actually this one.
It's an old book. It's from the time when the PC was a future concept, the Internet was research and tablets were science fiction. Yet it captures many of the conceptual problems we face when working in present day projects.

The reason I mention this book is because the authors have the most wonderful and simple problem break-down which goes as follow:

• What is the problem?
• What is really the problem?
• What is the problem, really?

Most projects only pay attention to the first bullet. Raise a defect, describe the initial problem. Done. Fix. Re-test. Done-done.

Instead of the "let's see how many defects we can close or down grade"-approach, try to apply the problem break-down method to your defects. Maybe not all of them. Maybe just the trivial ones. You know the most severe ones will get the attention anyway, so start from the bottom instead. 

Then you'll have the chance of understanding why so many defects are being reported. And just then you might also have some actions to do to prevent defects. That is when you understand your defects.

Load testing in the cloud

Problem: Scalability and setup can be problematic when setting up load and stress tests

Load testing is one of the tougher disciplines, not only does it require automated test execution, but it also requires high volumes of cases running every second. Scalability can easily become a problem, for your execution, and limitations in the infrastructure where the test is running dictates the test rather than your non-functional requirements.  

Solution: Take your load test to the cloud

We have quite some time used Microsoft Visual Studio & Azure to run distributed load testing. It is a test setup of a test controller with multiple test agents hosted in Azure, using Visual Studio for running the distributed test.

It is the test controller's responsibility to manage the test agents. The controller handles the execution of the test cases, by nominating who (ie which test agents) to perform the test and how the cases should be performed. It also makes sure to distribute the relevant test cases for a given test agent when the test started. When the test is finished, the controller collects the data from all test agents and this forms the basis of test results.

The test agent is responsible for the execution of the tests and simulation of a given number of virtual users. The test controller tells which tests should be performed and how many virtual users to be allocated for each test. Test Agent reports the status to the test controller during the test, and this information is used to generate the load test metrics.

For more information on test controllers and agents please see MSDN: http://msdn.microsoft.com/en-us/library/vstudio/dd648127.aspx

The trick is getting all the test controllers and agents setup, and then running the automated test on this setup. But once it is running there is really no limit to the number of test agents you can spawn to do your bidding. There is a nice demo on Youtube, showing running Web performance test and load test in Visual Studio 2010: http://youtu.be/yhkHtXcgWUc

Another thing that you might want to look into is the preview version of MS Visual Studio 2013 – It incorporates distributed load testing as a feature, cutting most of the setup of controllers and agents, making distributed load testing much more easily available.

For an overview of the new features of VS 2013 have a look at Brian Harry’s Blog: http://blogs.msdn.com/b/bharry/archive/2013/06/03/visual-studio-2013.aspx

Preview version of VS 2013 can be found here: http://www.microsoft.com/visualstudio/eng/2013-preview

Have a nice day & Happy testing!

/Nicolai