Combination testing strategies

Problem: Complete testing is impossible.

I had an interesting discussion about test of a critical feature. The discussion was on test coverage and completeness in the planned test. The argument was that this feature was of a criticality that required “complete testing”

 

Solution: Acknowledge that complete testing is not possible and apply combination testing strategies.

There are enormous numbers of possible tests. To test everything, you would have to:

·         Test every possible input to every variable.

·         Test every possible combination of inputs to every combination of variables.

·         Test every possible sequence through the program.

·         Test every hardware / software configuration, including configurations of servers not under your control.

·         Test every way in which the user might try to use the program.

No test satisfies all of above. How do we balance them? It is a matter of balancing the tradeoffs vs. risk, meaning that you get the highest coverage. I came across a nice slide set that I need to share, as it details different combination testing strategies:

http://www.slideshare.net/Softwarecentral/combination-testing-strategies

Happy testing!

/Nicolai

What is your mission?

Problem: Expectations vs. test team not aligned with the test team’s mission.

Do you know what your team’s mission is? Has this been aligned with expectations in the project and the customer? It is not unlikely that your answer is no and no, suggesting that there is a misalignment between your teams goals and the expectations of the project or maybe even the customer’s expectations

Solution: Make and communicate your team’s mission statement.

First step is to write a mission statement for the test team in the project. Do not confuse this with the mission statement that might exist for the testing services or QA department in general, this one needs to be for the project test team. There are numerous good guides to writing a mission statement – Google will help you with that.

Here are some bullets for inspiration when looking for your mission statement:

·         Find defects

·         Block premature product releases

·         Help customer make go/no-go decisions

·         Minimize technical support costs

·         Assess conformance to specification

·         Conform to regulations

·         Minimize safety-related risk

·         Find safe scenarios for use of the product

·         Assess and/or assure quality

·         Verify correctness of the product

Then it is time to starting communicating it. Start with the stakeholders closest to you, project management etc. and ensure that you at some point show it to the customer, and have a nice discussion on expectations to the outcome of the work being done by the test team. The mission statement is the means for a constructive dialogue on expectations and once you have come to an agreement with your peers it will give your team a clear purpose.

Print the mission statement and stick it on the wall – Make sure that it is catches the eye of those passing by, as it will lure them over for a chat about expectations and purpose. This will ensure that you are reminded about your purpose, and get a chance to reflect on the mission statement every now and then – Like plans, your mission statement might need revision in case expectations, stakeholders or deliveries change.

Happy testing!

/Nicolai

The curse of the notification e-mail...

I still remember one of the selling points from the Mercury sales pitch for TestDirector back in the days: ”The system is so clever! It will send you a notification Email with all the information you need every time that you get something assigned to you!” Since then I have received thousands of notification Emails from all kinds of ALM systems – Not all of them were especially clever or informative.

The notification Email is very a very popular feature, and widely implemented in various ALM and defect/issue management solutions. For those not very involved in the project they are a joy, as you do not need to visit the ALM tool unless you get a mail. For those in key positions in the workflow they are a pain, as you are already in the tool daily, and can find this information based on your dashboard or queries in the Tool. For those not understanding what they are used for, the mails become the curse that threatens the use of the ALM solution.

It is the administrator of the tool that carries the key to unleashing the curse, as (s)he can by default set the notification rules in a way that is not supporting the users. If (s)he at the same time adopts a  strategy where notification rules are global and not fixed to the user roles, then BAD things will happen.

The Emails are often not understood, containing lots of irrelevant data, or out of context, making them the equivalence of SPAM. It is wasting the recipient’s time, and makes him angry, fuelling the curse even further. Angry users are not very constructive, and those familiar with change management theory knows that it can provoke powerful negative reactions from stakeholders

 

This is why it is important to consider the usage of notification emails carefully before firing the mail-cannon at your organization. My advice would be to:

·         Identify and understand your stakeholders – Who will receive the mails?

·         Define who needs to be involved, who needs to be informed and what are the volumes of notifications.

·         Make sure all stakeholders are appropriately informed about the ALM notifications and ensure that your offer them a way to modify the mail subscriptions to fit their needs.

·         Consider doing daily, weekly or monthly notification roundup mails (if feature is available in tool).

·         Ensure that the template used for the mails has only the required information needed, and not everything.

Alternatively you are likely to experience some of the following:

·         Users applies mail autofiltering, sending the notification mail directly in the archive (or deleted items)

·         Users complains that ALM system is a pain, and stops using the tool.

·         Users gets stressed by all the mails they get.

·         Users starts to forward notification emails, disabling the ALM workflow in the tool.

·         Users looses track of other items in their crowded inbox.

All of the above is waste, which is injected and maintained directly in your development cost – Not as clever as the sales pitch originally stated…

Happy testing!

/Nicolai

Tips for registering defects

We have just been through a couple rounds of testing, and gathered a lot of observations. Some of these are not defects, they are the result of tester wish-listing, misunderstandings, wrong test cases etc. This is however as expected, and as always calls for a bug triage and refinement BEFORE any fixing and distribution of the defects happen – In other words, no defect management before you are actual sure that it is a defect.

Luckily for us, the testers have been very careful when raising an issue – Something that we benefit from now, saving vast amounts of time in the analysis. Despite the testers being new to testing, they have reported the observations in a way that facilitates future analysis. Simply by following our tips for issue reporting!

There  is little new in the tips for issue reporting below for all you professional testers, but for business reps who are invited to join a project as testers, everything is new, and that is when you will benefit from guides that explains key areas of your test process. We made such a guideline, consisting of a manual for using the test tool – Stripped for anything but what the testers needed. And a list of advice to help filing the issues while testing.

Make sure that you nurse your testers, so they understand their role and the expectations towards their deliveries – They become a powerful asset if you do.

 Happy testing!

/Nicolai

Tips for registering issues:

Reproduce the issue before filing an issue report: Your issue should be reproducible. Make sure your steps are robust enough to reproduce the issue without any ambiguity by someone who is not you. If the issue is not reproducible every time you can still file an issue mentioning the periodic nature of the bug.

Report the problem immediately: If you found any issue while testing, do not postpone the detailed report to later, instead write the issue report immediately. This will ensure a good and detailed bug report. If you decide to write the bug report later on, then chances are that you miss the important steps in your report.

Spend 2 extra minutes on the description: When writing a description please adhere to the following structure, to ensure that sufficient information is captured:

·         Reproduce steps: Clearly mention the steps to reproduce the issue, to facilitate reproduction of the issue by developers and others who will work on fixing the problem.

·         Expected result: How application is expected behave on above mentioned steps. Add a reference (if known) to the requirement that appears to be violated.

·         Actual result: What is the actual result on running above steps. Include a screenshot of the scenario, or clear detailing on test scenario, including test data used.

Read issue report before hitting SAVE button
Read all sentences, wording, steps used in bug report. See if any sentence is creating ambiguity that can lead to misinterpretation. Misleading words or sentences should be avoided in order to have a clear bug report, and ease communication with the receiver.

Assumptions – The mother of all evil?

Yesterday we were discussing some topics related to a delivery, and I became aware of something our counterpart was saying things like: “We assume…”, “It is safe to assume…”, “You might be in a position to assume…” – That led me to lose faith in the plan that they had prepared.

In my book assumptions are a constant source of risk, due to the fact that they represent something that is unknown and hidden under a blanket of guessing. They are however easy to make, simplifies things and offers a shortcut in situations where unknown variables are encountered. In other words a necessary evil for most of the things we do and when made consciously they might even serve you well. If assumptions are injected unconsciously bad things might happen on the basis the risk you indirectly accept.

This corresponds (more or less) with the dictionary definition of assumption: “Accepted cause and effect relationships, or estimates of the existence of a fact from the known existence of other fact(s). Although useful in providing basis for action and in creating "what if" scenarios to simulate different realities or possible situations, assumptions are dangerous when accepted as reality without thorough examination”

Back to the plan under discussion the other day… With all the assumptions aired while walking through the plan I had to ask the author about the foundation on which it was build. My claim is that it could hardly be estimates, as many of the activities was based on assumptions, it would have to be guesstimates, or in other words inaccurate estimates – This argument caused quite a stir.

We ended up agreeing that the plan was fine, but all assumptions that were related to activities on the critical path of the project had to be documented. These assumptions had to be proved right or dismissed with adequate re-planning as a consequence. Until this point in time they where they were considered risks. It was hard work identifying the assumptions, but gave a good picture on risk vs. critical path.

Conclusion: Assumptions are not the mother of all evil but be very conscious when operating on assumptions. You cannot avoid them, but rest assured that assumptions are the equivalence of unmanaged risk if you are not, especially if you build critical plans on assumptions!

Happy testing!

 

/Nicolai

Choosing the right approach for the test plan

Following my game of who-is-who last week I continued to think about the questions (or test cases) used in the game. I was wondering if a traditional test approach would work in a setup like the one encountered in who-is-who.

 

So I tried preparing the game just like I would if I was following the approach suggested by the V-model:

·         I listed all questions I would ask (test case preparation)
·         I ordered the cases in a sequence (fixed test plan)
·         And then I played the game (test execution)

 

Result: I lost miserably – After several games where I was tweaking my test plan I did not win a single game. The reason was, that I could not change the test plan based on experience gained while playing, meaning that some of my questions have little to no impact.

 

Then I changed strategy – Experience driven test execution following this approach:

·         I listed all questions I would ask (test case preparation)
·         I asked questions based on experience (no fixed test plan)
·         And then I played the game (test execution)

 

Result: I started winning, but I spotted some problems in the approach – I found situations where test coverage was insufficient, and lacking test cases I was in situations where my poor test coverage resulted in me loosing, or taking the long route to the answer I was looking for.

 

Conclusion: Changing strategy again to a charter-driven, experience based test would likely have been the best approach for a game like this. There are a lot of strategies when testing, picking the right one is the hard part. This suggests that the test needs to be driven by more than a good question or test case – That being the ability to ask the right question at the right time.

 

Happy testing!

 

/Nicolai

Asking the right question?

I played a game of who-is-who yesterday. The game is all about eliminating options until you are able to guess who your opponent is. In other words: It is all about probing for specific information, that will give you a complete picture of your opponent – Much like testing an application, where you’re verifying and validating until you reach agreed level of knowledge.

 

Probing for information is about asking questions. Poor questions and poor questioning techniques interfere with the information gathering by creating confusion and misinformation, just like poor test cases will lead to false positive and negative results.

 

Question: So thinking about questions like test cases and visa verse I started wondering if questioning techniques for human communication could be applied when writing test cases…?

 

To answer a question like that I had a look at questioning techniques from the HR & communication world, and is seems that there are many takes on how to ask a good question, and interesting enough several takes on inefficient questioning. Less effective questioning consists of: Closed ended questions – Answered by Yes or NO and Leading questions – Containing the answer imbedded in the question. A test step is answered binary (pass or fail) and contains the answer in the step as expected result, suggesting that the traditional test case is using less efficient methods for gathering information?

 

I tend to think that questioning a person and testing a machine are miles apart. When questioning a person you have the benefit of verbal and nonverbal communication. When testing a machine you get observations that need to be checked versus an expected result. That is why you need to ask a closed ended question in your test case, and have the answer you are looking for in the case – Something that (according to the communication experts) is a NO-NO when talking to people.

 

Consider this example:

 

 

Human interpretation	Machine interpretation
There is no definition of how to identify a bitch. Outcome is complicated, Mr. Jackson’s and my perception of ‘the bitch variable’ might differ, meaning that answer will never be unambiguous enough to get a result that would work in a game of who-is-who.	Check vs. variables will be BitchVariableExists = Yes or No and BitchVariablePopulated = Yes or No? Outcome is simple, either variable exists or not, and if it exists it can be either populated or not.

 

This means that the test case prepared by Mr. Jackson would work if run vs. a machine, but not in an analogue game of who-is-who.

 

Conclusion: Questioning techniques for people communication are not necessarily a benefit when writing test cases, as these must be unambiguous, hence have to rely on the closed ended questions. Questioning techniques are however paramount when you gather from your peers and human test oracles information for writing your test case.

 

Happy testing & have a nice weekend!

 

/Nicolai