Problem: Too many parameters to test all combinations thoroughly!
Ever been in a project that had to develop and test something that involved a large number of input variables as part of a business process?
Then you also know that testing all combinations is impossible and if you are that test analyst or test manager dealing with the project it will give you sleepless nights.
Solution: Apply pairwise or all-pairs as part of your test preparation effort!
Apart from the fact that the math behind is a bit tricky to understand the good part is that this actually works - in practice. A lot of tools exist - free to use to support the process of actually figuring out which combinations to test in order to get high coverage.
There is an excellent site that holds a list of updated tools. My favourites include:
CTE-XL because it has a GUI and it has the potential to create "test case input" that can be exported to Quality Center. It takes more time to setup, more time to understand the UI and usage, but it has great potential to do graphical overviews of complex dependencies.
PICT has a new "GUI" because somebody decided to do an Excel sheet that interfaces with the tool. PICTMaster can be downloaded from Sourceforge. Well documented by the way!
Pairwise is good but it doesn't replace the business analyst who has the experience and knowledge to pinpoint import business test scenarios. However you will get good input for estimation and in some cases input for pinpointing unclear or missing requirements. More on that in a later post.
Ever been in a project that had to develop and test something that involved a large number of input variables as part of a business process?
Then you also know that testing all combinations is impossible and if you are that test analyst or test manager dealing with the project it will give you sleepless nights.
Solution: Apply pairwise or all-pairs as part of your test preparation effort!
Apart from the fact that the math behind is a bit tricky to understand the good part is that this actually works - in practice. A lot of tools exist - free to use to support the process of actually figuring out which combinations to test in order to get high coverage.
There is an excellent site that holds a list of updated tools. My favourites include:
- PICT from Microsoft
- CTE-XL from Berner & Mattner
CTE-XL because it has a GUI and it has the potential to create "test case input" that can be exported to Quality Center. It takes more time to setup, more time to understand the UI and usage, but it has great potential to do graphical overviews of complex dependencies.
PICT has a new "GUI" because somebody decided to do an Excel sheet that interfaces with the tool. PICTMaster can be downloaded from Sourceforge. Well documented by the way!
Pairwise is good but it doesn't replace the business analyst who has the experience and knowledge to pinpoint import business test scenarios. However you will get good input for estimation and in some cases input for pinpointing unclear or missing requirements. More on that in a later post.
I remember a project where we were modeling containers for a shipping solution, the calculation on number of test cases needed to cover all combinations of the variables a container could have was 2.3 million. This was interesting as it would take forever to test - Even using automated test scripts.
ReplyDeleteWe used the calculation for an interesting dialogue with the client on how to regression test the solution, without covering all variables, and ended up with 50 or 60 cases that had good coverage while being executable every time we made a release candidate.
/Nicolai