Friday 25 March 2016

Top 10 audit topics for test cases

So, a quarter of 2016 has passed - almost - and your projects are probably busy, summer deadlines approaching, lots of testing to take care of at the moment. And then - the quiet summer holiday period is also within sight. Maybe its worth considering what to do during those hot but quiet weeks this year.

An idea could be to prepare for an audit of the test cases. A bit of structured house keeping. So here are a top 10 suggestion for things to look after to get rid of crap and dig out the gems for future testing purposes.


  1. Discover and discard. Have a critical look at test cases that are not going to create future value. My own favorite - test cases with zero (0) step. The ones that were created with the best intentions but for some reason never made it beyond "design". Delete them or archive them in a way that they don't make any noise in future reporting
  2. Find the gems. Look at your test cases from the opposite angle. Which test cases were the "favourites" in recent projects and test runs. Identify those, especially the ones that actually are known to have detected defects and make sure they are documented and accessible for future testing purposes.
  3. Filter through the remaining 80%. OK, so you're now off to a good start, blanks gone and gems identified - "the rest" is just remaining. Unless you have a very strict approach to test case documentation you should probably be left with the 80% of the test cases you started out with. What to do now? For starters you could take a look at business criticality - which test cases do actually make sense to keep for future testing from a strict business point of view. They should be the next in the "save-for-later-bucket".
  4. Chop through duplicates and variations of the same theme. Chances are you don't need six test cases that aim at testing the same functionality in slightly different ways. If you do consider some kind of bundling. If you don't, make a choice and clear away the remaining test cases. 
  5. Find the blank spots. This is where it gets interesting. If you have spent the time so far going through documented test cases you should by now also have a few "eureka moments". You should have a few Post-it notes or a short list of things that should be somewhere in the test documentation but is no where to be found. Hunt down those responsible and find out what actually happened. And finally make sure that the blanks are filled with adequate documentation where needed.
  6. Traceability. Don't start re-doing business requirements now. It is too late. Instead make sure that you have some kind of adequate, high level overview and that you in that way maintain the overview of which areas have test coverage and which areas are left uncovered. Chances are that there is way too much work to be done, but at least you now know where the potential gaps are for future projects.
  7. Take a break. No, not that kind of break. Instead spend a short time bothering the DevOps, support team, first line support, the incident manager - or all of them depending on who's actually available now. Don't bother them with long lists about test cases and requirements coverage - instead have their input in terms of bug and errors detected during recent time and then do the analysis if it is feasible to include some of that into future testing. Or have a bucket list to present to the test analysts for later.
  8. Check the consistency of regression test packages. Are you adequately covered with a selection of test case packages for different regression test purposes? Is your smoke test good? Like in robust, focused and bullet proof. Can it be run in a short time? Are you genuinely happy with the efficiency of the test cases for smoke testing purposes?
  9. Test data pitfalls. By now you should be somewhere between concerned and desperate if you have covered bullets 1 through 8. There is a lot of work to do. Now consider if you have the right test data to support testing. Don't, however, do another 8 point list for test data because then you will be stuck with a lifetime project for that alone. Instead look at test data from a helicopter perspective. Remember this is the audit, it is not a "fix-the-world"-project. Where are the big loopholes in terms of test data, given that you now know which tests are actually creating future value for you and the organisation you are with? A favorite topic is always testing that had to be de-scoped due to test data issues. Some types of test data are difficult to work with in real life. Like data that is consumed due to one-time usage only. Maybe it is the right time to think about possible solutions now. Or simply document risk and impact a bit better given that your knowledge should be better.
  10. Sanity check. Lean back and enjoy a world that is less messy and better understood. And therefore a bit easier to communicate. Last topic on the list is to do a short sanity check. Can you actually report to project stakeholders based on the test case packages that you have decided to keep? Is the coverage sufficient? Where are the weak areas? Are there any ways to strengthen the weak areas? Improvements? Priorities? Congratulations. You now know what to include in your project activities for the remaining part of 2016 - and beyond.