The project was delayed. Classical V-model approach with a couple of rounds of system test and system integration test at the end of the plan. Development turned out to be even more problematic for a number of reasons mostly attributed to the integration betweeen the systems of an internal and an external supplier.
The test team used the waiting time for preparation so when the day came and the environment was smoke tested all pressure lay on the team - and they were ready as ever. Since the additional preparation effort paid off testing managed to stay within the plan for the first round of testing. Thumbs up.
Bugs were fixed, testing started once again and then some more bugs were discovered. An additional round of testing was agreed and planned and just before that was about to start - disaster was discovered.
The configuration of the test environment was flawed. Big time. All the integrations were set up correctly, technical users and other trivial stuff worked but the data and configuration of static data was wrong. All test results were invalid. All bugs were false positives. Everybody, including the project manager, had been so eager to get testing started that nobody had bothered questioning the baseline of the test data configuration. And the initial smoke test had proven that the environment was fully operational for testing - but not that data was correct.
Two weeks wasted. All energy sucked out of the project.
Prepare and describe the test data baselining test activities! Especially if test data is crucial to your test.
- Identify crucial test data as early as possible
- Understand how to validate that data in the environment before testing starts
- Perform a review to get a second opinion
- Plan a smoke test that focuses on proving that data is present in the desired state in the test environment
- Execute the smoke test
- Analyse and understand test result.
Proving that test data is correct is one of the most tricky tasks for a test team. But a little effort can determine whether it feels right or wrong to go ahead with the test.