Pairwise testing

Problem: Test cases covering multiple variables can be tricky to create

Lately we have been testing workflow rules. This calls for test cases that covers many variables, and as the number of variable goes up, so does complexity. We started the test specification in front of a decision tree, and quickly realized that writing the cases manually was too time-consuming, not to mention risky as the number of combinations was too much for us to cope with.

Solution: Apply all-pairs testing for the test case creation, and a tool to do the pairing.

There are many tools for creating the test cases, and the place to visit is www.pairwise.org Here you will find a tool that does the job, no matter if you prefer command prompt, text driven or GUI driven tools that can create nice graphical models of the test.

I like it simple, so I used PICT, a command prompt driven tool that takes text files as input and delivers text files as out put. Easy-peasy, just put your variables in a file, and transform that into a list of test cases that you can use for your test. For more information on PICT I suggest that you read the help-file found here: http://www.amibugshare.com/pict/help.html

Many of our test cases are parameter driven in MS test manager, meaning that it is possible to copy the result from the result file directly into the parameters into the test case. The only precondition is that you make sure that the columns in the result are in the same order as the parameters in MS test manager.

Consider the technique and tools for input test scenarios.

Happy testing !

/Nicolai

Are you ready for testing?

Problem: Assumptions about test readiness will ruin your day

Like any other project activity test execution relies on agreements with stakeholders, preparations and deliveries to take place before the test can start. I have seen quite a few test runs being foiled by invalid assumptions and unknown status.


Solution: Test Readiness Review

Fire up your entry criterias in a Test Readiness Review before running the test. Either as an informal or a formal test readiness review. This is how I usually do it:

All my test readiness reviews are meetings where stakeholders are invited to have a talk about the entry criteria for the test that is about to be executed. The level of formality differs depending on the test that is being done. Internal tests that only involve internal stakeholders will be informal reviews, where external test runs with external stakeholders will be very formal.

Informal Test Readiness Review

Done in relation to System and System Integration Test (to internal systems)
When: A couple of days before test start.
Length: max 30 min
Who: Test team, key players in development org (like build manager, lead dev, project manager etc.)

I call for a meeting where only preparation is to bring a cup of coffee, and the agenda is discussion of the preconditions / entry criterias for the test to come. I gather information about readiness up front, making a shortlist of questions / actions to be discussed, but majority of the meeting is information about who does what.

Formal Test Readiness Review

Done in relation to System Integration Test (to external systems) and acceptance test
When: One week before test start.
Length: 1 hour+ (depending on the size of the test project)
Who: Test teams, key players in delivery organizations (like project managers etc.)

I call for a meeting where agenda states the entry criterias and responcibilities (from test plan) and preparation for the participants is to give status on their readiness. The agenda is a short presentation of the test to come and readiness status from all participants.

Outcome of the meeting is a status/action list that states exactly what needs to be done by who before test can start.

If your test readiness review raises a concern about being ready on time, then you have time to proactively doing something about the problem before the test is running. At the very least you can report the risk to the project owner.

The trick is to raise awareness about the test to come – This is often forgotten in the rush of completing the coding. I recommend that the meetings are called the second that test plan is approved, in order to reserve the time in peoples calendars and the create visibility about the milestone that delivery to test is.

Happy testing!

/Nicolai

EDIT: You can see my test readiness checklist here: http://testing4tw.blogspot.dk/2014/01/test-readiness-review-checklist.html