Combination testing strategies

Problem: Complete testing is impossible.

I had an interesting discussion about test of a critical feature. The discussion was on test coverage and completeness in the planned test. The argument was that this feature was of a criticality that required “complete testing”

 

Solution: Acknowledge that complete testing is not possible and apply combination testing strategies.

There are enormous numbers of possible tests. To test everything, you would have to:

·         Test every possible input to every variable.

·         Test every possible combination of inputs to every combination of variables.

·         Test every possible sequence through the program.

·         Test every hardware / software configuration, including configurations of servers not under your control.

·         Test every way in which the user might try to use the program.

No test satisfies all of above. How do we balance them? It is a matter of balancing the tradeoffs vs. risk, meaning that you get the highest coverage. I came across a nice slide set that I need to share, as it details different combination testing strategies:

http://www.slideshare.net/Softwarecentral/combination-testing-strategies

Happy testing!

/Nicolai

Pairwise testing

Problem: Test cases covering multiple variables can be tricky to create

Lately we have been testing workflow rules. This calls for test cases that covers many variables, and as the number of variable goes up, so does complexity. We started the test specification in front of a decision tree, and quickly realized that writing the cases manually was too time-consuming, not to mention risky as the number of combinations was too much for us to cope with.

Solution: Apply all-pairs testing for the test case creation, and a tool to do the pairing.

There are many tools for creating the test cases, and the place to visit is www.pairwise.org Here you will find a tool that does the job, no matter if you prefer command prompt, text driven or GUI driven tools that can create nice graphical models of the test.

I like it simple, so I used PICT, a command prompt driven tool that takes text files as input and delivers text files as out put. Easy-peasy, just put your variables in a file, and transform that into a list of test cases that you can use for your test. For more information on PICT I suggest that you read the help-file found here: http://www.amibugshare.com/pict/help.html

Many of our test cases are parameter driven in MS test manager, meaning that it is possible to copy the result from the result file directly into the parameters into the test case. The only precondition is that you make sure that the columns in the result are in the same order as the parameters in MS test manager.

Consider the technique and tools for input test scenarios.

Happy testing !

/Nicolai

Too many combinations to test...

Problem: Too many parameters to test all combinations thoroughly!

Ever been in a project that had to develop and test something that involved a large number of input variables as part of a business process?


Then you also know that testing all combinations is impossible and if you are that test analyst or test manager dealing with the project it will give you sleepless nights.


Solution: Apply pairwise or all-pairs as part of your test preparation effort!


Apart from the fact that the math behind is a bit tricky to understand the good part is that this actually works - in practice. A lot of tools exist - free to use to support the process of actually figuring out which combinations to test in order to get high coverage.

There is an excellent site that holds a list of updated tools. My favourites include:

• PICT from Microsoft
• CTE-XL from Berner & Mattner

PICT because it is command line, simple to download and uses input from at text file. It can hardly get any more simple, but command line will also be less appealing to some users. The learning curve is something like 10-15 minutes and you'll have your first result.

CTE-XL because it has a GUI and it has the potential to create "test case input" that can be exported to Quality Center. It takes more time to setup, more time to understand the UI and usage, but it has great potential to do graphical overviews of complex dependencies.

PICT has a new "GUI" because somebody decided to do an Excel sheet that interfaces with the tool. PICTMaster can be downloaded from Sourceforge. Well documented by the way!

Pairwise is good but it doesn't replace the business analyst who has the experience and knowledge to pinpoint import business test scenarios. However you will get good input for estimation and in some cases input for pinpointing unclear or missing requirements. More on that in a later post.

 

Input testing checklist

Problem: Input testing is not negative enough

One of our developers had an open book on his table. It had a catchy title for chapter 10: All Input Is Evil! The book is Writing Secure Code, Second Edition by Michael Howard  & David LeBlanc. I got rather curious and read the chapter, which was on the problems related to defensive coding versus input. It outlined two rules; “All input is evil until proven otherwise” and “data must be validated between untrusted and trusted environments”

That made me think – Why not make some kind of generic test case for input tests, that we could use repeatedly. From a test perspective, this case should be useable for manual and unit testing, not directly, but as a kind of checklist.

 

Solution: Equivalence Partitioning, boundary value analysis and positive/negative test cases for input testing.

We deal a lot in web-based applications that are driven by user input. Users are creative, some more than others, but at some point the user will violate all rules either by mistake or on purpose. Handling input like that needs to have been subject to test before our product leaves the code factory.

A generic test case should test for the following:

• Check validity – Input of logical values and illogical values, this is where positive and negative equivalence patitioning is extremely important.
• Check ability to handle all Unicode chars – Check UTS#18 Unicode Regular Expressions §1.2.1 for inspiration on the Equivalence Clases that needs to be tested. Remember country specific chars – I live in Denmark, we have letters like æøå & ÆØÅ to test for…
• Script injection – Input scripts, tags from SQL, HTML og XML.

Example using the above generic test checklist.

Consider the following input source: Namefield [2-28 char] a-z only






This field would require a minimum of 11 cases to cover the input (not including Script injection).

• Testing field length alone would take 6 cases: x<2, x=2, 2<x<28, x=28, x>28, NULL (example cases: x= 1, 2, 14, 28, 32, NULL)
• Testing alfa char [a-z] would take 5 cases: x in [a-z], x in [A-Z], x in [0-9], x in [spc. char], x=[space] (example cases: x=Eric, ERIC, 1234567890, !@#$%^&*()_+<>?,./;':"☺☻♥♦♣♠•◘○, [space])
• X cases for script injection (example cases: x=<ahref=www.google.com>, x=<br>).

That is many cases for a simple field, and most of these are easily forgotten when designing tests, that be unit-, automated- or manual test cases. I would like to encourage you to think about those negative Equivalence classes and boundary values, as they are 10 times more likely to hide a bug than the test cases that probe the positive side of things, simply because IT professionals forget that “All input is evil!”

 
Have a nice day & Happy testing!

/Nicolai