Tuesday, 10 May 2016

Well written test cases – or not. A few thoughts on test design

One of those recurring discussions when preparing for test is “how much detail is required in the test cases?. Those SME’s participating from the line of business know their business, they know the business rules and they know all the “sore spots” – those alternative test cases that need to be run in order to find some of the nasty bugs. So why in the world should they spend the time writing detailed test cases when they can basically test from the library within their memory?

On the other side of the table there is usually a single test manager trying to argue for “real” test cases. Those documented test cases that follow the basic guidelines from IEEE 610 with real preconditions, expected values and a nice list of things to do when a tester is executing the test case.

And that’s where the discussion usually ends. SME's reluctant to spend all their energy writing long test cases and test managers eager to have good project documentation.
There is logic to avoiding the massive overhead of doing detailed test cases in some projects when the right SME participants are in the project. And there is another reason to do detailed test cases in most projects – compliance. The whole idea with detailed test cases about having a baseline that can be used for consistent runs and reruns of test, reproducing bugs and for documenting “what actually happened” is enough to justify the effort. And that brings us back to how detailed the test cases should be.

In most projects it might be enough to have test cases with a few steps and some check points to those steps. Why bother doing step-by-step test cases when the testers build up knowledge about the application in a short while? Test cases that are only headlines are not sufficient simply because they open up for too much room for interpretation effectively making it impossible to report on test coverage and progress for that sake.

One thing that seems to work when the discussion about the level of details comes up is the “good example”. This should be prepared by the test manager and used as a template for those working with test case documentation. With solid arguments about why a certain level of details is necessary. Reproducibility, compliance, internal guidelines, company policy, business criticality are all valid reasons to ask for a certain level of detail for test cases. The test manager should drive the process towards a common agreement of detailed test cases and the understanding of why this level is necessary.