Standard Risk List for Projects

Problem: Risks are cumbersome to identify, and requires grooming before discussion can take place.

Working with risks require that they are well documented and made accessible for the organization to absorb and work with. Grooming the risks takes time, allowing some of the risks to grow into issues, or slip under the radar and avoid getting mitigation tasks in the plans. This means that early identification of risks will make test planning and risk management a lot easier.

Solution: Start with a standard risk list, and build your planning and risk management on that.

In many projects it is the same problems (or variations of same) you will encounter. So you might as well look at the usual suspects. I did that a while back, and concluded that I had to make a standard list to use for input to test plans and risk meetings in the projects I participated in.

Given that test management is my game, the risks focus on test issues and impact on quality, and to make them easily available I divided them into these categories:

Resource related risks:

• Use of undedicated resources
• Poor Productivity in long projects
• Wrong or missing test resources
• Employee Turnover

Process related risks:

• Big bang releases
• Overlapping project phases & activities
• Configuration and Change Management
• Risk Management not being done

Delivery related risks:

• There will be a delay in the release of the product
• Gold plating
• Stakeholders and expectations not managed

Documentation related risks:

• Requirements not complete & contain assumptions
• Requirements Inflation
• Specification Breakdown
• Compromising on designs
• Technical risks
• Inter plan dependencies
• Scope is not closed.
• Test foundation not in place
• Test environment & Test data not in place
• Lack of Test Tool usage

Each detailed with the following information: Risks, Event, Possible causes, Effects, Possible measures to facilitate discussion and mitigation.

You can find my standard risk list for projects here.

Note: This is not an invite to stop inventing risks, and run risk management, but it is a fast-track to getting risk management going early in projects. It is an invitation to get the test and QA related risks on of the list and help you write the best test plan ever.

Have a nice day & Happy testing!

/Nicolai

*** EDIT: I came across this very comprehensive list of risks in SDL, something that you might find handy: http://www.dtvisiontech.com/2014/11/software-risks-in-software-development.html 

Visual Studio Test Tooling Guides

I need to share this, as it is a real treat for all working with test in MS Visual Studio. We discovered this by accident, while working with coded IU tests this week.

http://vsartesttoolingguide.codeplex.com/

In here you will find comprehencive guides to everything related to test in the MS Visual Studio / Test Manager universe.



It contains the following guides, which are all worth a look:

• Coded UI Guide
• DevOps bug resolution using IntelliTrace
• Better Unit Testing with Microsoft Fakes
• Microsoft Test Manager Guide

Enjoy & Happy testing!

/Nicolai

Automation Considerations

I came across this today:

 From http://xkcd.com/

It underlines the need for Return on Investment discussions and measurements when working with automation. There are so many factors that can kill a good automation case, where running maintenance and gold plating of the solution are some.

I wrote something on automation RoI a while back, and the points from above comic really underlines the points about deploying RoI and proof of concept as drivers for evaluating the results of implementing automated tools.

The previous post on test automation return of investment can be found here:

http://testing4tw.blogspot.dk/2013/09/test-automation-return-of-investment.html

 

Have a nice day & happy testing!

/Nicolai

Test Readiness Review Checklist

A while back I wrote about using test readiness reviews as part of test governance toolbox : http://testing4tw.blogspot.dk/2013/12/are-you-ready-for-testing.html

I came across my test readiness review checklist, and thought I would share it:

Deliveries to test:

·        Is feature list complete and updated?

·        Are some features not ready for test?

·        Have previous test phase been completed?

·        Does the delivery contain known errors / shortcomings?

Test documentation:

·        Are test cases ready and reviewed?

·        Have test cases been set up in test tool?

·        Are test cases involving 3^rd party stakeholders discussed and agreed?

·        Has test log and reporting templates been prepared for the test?

·        Has invites been sent for test readiness review, and test status meetings for the test to come?

Test Data and Environment:

·        Is test data load needed, planned and detailed?

·        Is test data requirements in place and supported in test environment?

·        Is the environment configured according to expectations?

·        Has deploys of test builds been scheduled?

·        Are there any know environment outages (upgrades etc.) in the test period?

 

Aside from above you need to obtain a statement of: “We are ready and able to execute test from date X” from all stakeholders in the test phase to come.

Have a nice day & Happy testing!

/Nicolai

Test Process Improvement

Problem: Process Improvement without clear priorities and goals will fail

Many organizations do not have sufficient transparency in the test processes deployed and the result of this investment, this makes it hard to deliver the expected result to the business. Much like hiking you require a map, a compass, your destination and most importantly your current location – Without known location and clear goals you will get lost, no matter whether you are hiking or doing process improvement.

Solution: Test Process improvement based on maturity assessments.

Why do maturity assessments?

The test maturity assessments are done in order to guide, develop and prioritize the improvement initiatives in the organization. It allows you to work on your initiatives on a strategic, tactical and operational level in a controlled manner.

Strategic	·        Create common foundation and visibility on vision and goals for test and QA. ·        Create understanding and commitment for test and quality in the organization. ·        Get insight on cost of quality and RoI in the projects ·        Serve as a communication and governance tool.
Tactical	·        Set priorities for the initiatives. ·        Guide the initiatives in a controlled manner. ·        Improve process and competences.
Operational	·        Optimize use of resources ·        Reduce waste in projects ·        Streamline test effort and approach.

When is a maturity assessment needed?

·        If there is an increased need for streamlining of effort and process. E.g. when complexity increases in IT, dependencies on key resources, use of external resources, working with external partners, offshoring of development and test.

·        Deteriorating quality in the deliveries. E.g. Lots of defects in production, cost escalation, lack of confidence in deliveries, economic consequences to the business.

·        There is a need for information to do decision-making, prioritizing new initiatives etc.

·        Projects are high risk and of a critical nature.

What tools are available? What are the methods?

There are a lot of tools and methods you can use to do maturity assessments, such as TPI, TMMI, CMMI, ISO etc. It really doesn’t matter which one you use as long as it sets the boundaries and priorities for the effort and secure a holistic approach to the improvement activities.

What does it take to get started?

I use TPI (Test Process Improvement) developed by Sogeti, it consists of a field test tool that allows you to make the assessment based on interviews of key people in the organization. The result generates an overview of the situation, and gives a framework that allows a controlled and prioritized approach to improving quality.

The details about how-to is described in the book “TPI NEXT – Business Driven Test Process Improvement“ (ISBN 90-72194-97-7). Read the book and play with the tool, it is a small investment that gives a lot in return. Try answering the questions in the questionnaire, and you will get an instant result.

The TPI materials can be found here: http://www.tmap.net/en/tpi-next/downloads

A word of advice if you are the slightest in doubt about if you can answer positive to a question then it should be no, and that translates into something that you should look into. Using the questionnaire in projects, interviewing people like project manager, test manager, lead developer and business test persons will in the matter of hours give you a clear view of ‘state of the nation’. For starters, you only need to look at the controlled level, which should give you plenty of observations to work on.

Happy testing!

/Nicolai

 

Pairwise testing

Problem: Test cases covering multiple variables can be tricky to create

Lately we have been testing workflow rules. This calls for test cases that covers many variables, and as the number of variable goes up, so does complexity. We started the test specification in front of a decision tree, and quickly realized that writing the cases manually was too time-consuming, not to mention risky as the number of combinations was too much for us to cope with.

Solution: Apply all-pairs testing for the test case creation, and a tool to do the pairing.

There are many tools for creating the test cases, and the place to visit is www.pairwise.org Here you will find a tool that does the job, no matter if you prefer command prompt, text driven or GUI driven tools that can create nice graphical models of the test.

I like it simple, so I used PICT, a command prompt driven tool that takes text files as input and delivers text files as out put. Easy-peasy, just put your variables in a file, and transform that into a list of test cases that you can use for your test. For more information on PICT I suggest that you read the help-file found here: http://www.amibugshare.com/pict/help.html

Many of our test cases are parameter driven in MS test manager, meaning that it is possible to copy the result from the result file directly into the parameters into the test case. The only precondition is that you make sure that the columns in the result are in the same order as the parameters in MS test manager.

Consider the technique and tools for input test scenarios.

Happy testing !

/Nicolai

Are you ready for testing?

Problem: Assumptions about test readiness will ruin your day

Like any other project activity test execution relies on agreements with stakeholders, preparations and deliveries to take place before the test can start. I have seen quite a few test runs being foiled by invalid assumptions and unknown status.


Solution: Test Readiness Review

Fire up your entry criterias in a Test Readiness Review before running the test. Either as an informal or a formal test readiness review. This is how I usually do it:

All my test readiness reviews are meetings where stakeholders are invited to have a talk about the entry criteria for the test that is about to be executed. The level of formality differs depending on the test that is being done. Internal tests that only involve internal stakeholders will be informal reviews, where external test runs with external stakeholders will be very formal.

Informal Test Readiness Review

Done in relation to System and System Integration Test (to internal systems)
When: A couple of days before test start.
Length: max 30 min
Who: Test team, key players in development org (like build manager, lead dev, project manager etc.)

I call for a meeting where only preparation is to bring a cup of coffee, and the agenda is discussion of the preconditions / entry criterias for the test to come. I gather information about readiness up front, making a shortlist of questions / actions to be discussed, but majority of the meeting is information about who does what.

Formal Test Readiness Review

Done in relation to System Integration Test (to external systems) and acceptance test
When: One week before test start.
Length: 1 hour+ (depending on the size of the test project)
Who: Test teams, key players in delivery organizations (like project managers etc.)

I call for a meeting where agenda states the entry criterias and responcibilities (from test plan) and preparation for the participants is to give status on their readiness. The agenda is a short presentation of the test to come and readiness status from all participants.

Outcome of the meeting is a status/action list that states exactly what needs to be done by who before test can start.

If your test readiness review raises a concern about being ready on time, then you have time to proactively doing something about the problem before the test is running. At the very least you can report the risk to the project owner.

The trick is to raise awareness about the test to come – This is often forgotten in the rush of completing the coding. I recommend that the meetings are called the second that test plan is approved, in order to reserve the time in peoples calendars and the create visibility about the milestone that delivery to test is.

Happy testing!

/Nicolai

EDIT: You can see my test readiness checklist here: http://testing4tw.blogspot.dk/2014/01/test-readiness-review-checklist.html