A few thoughts about risk

We've been around this topic a number of times - risk based testing. Another Danish blogger did a short post about this (in Danish) just before the summer holidays.

The point is right - base the test approach on the identified risks. There are numerous problems in that approach. A few mentioned below.

Customer input

The "stakeholders" who should be able to list and prioritise risks hardly know what they've bought or ordered. Most organisations or customers have little idea about what's going to hit them when they embark on projects where risk based testing could be of good use. They are about to participate in the expectation roller coaster ride where each day will bring new challenges and decisions forward and on top of that they have to prepare their own organisation and their customers for the changes they've ordered. They are not used to working with the fluffy term of "risks" and being the test manager trying to facilitate them is no easy task.

The professional testers

This group of project participants have to be kept on a short leash when it comes to risks. Depending on their interest, experience and beliefs they can carpet bomb any risk session in a matter of minutes. Most of them with very relevant input, but too often it is input that is way outside what the project is capable of dealing with - even with lots of resources (time, hands and money).
That said, testers are usually your best chance of success when it comes to qualified risk input based on their experience but put on your angry test manager hat and cut away any risks that are improbable or impossible to deal with.

Development and service organisation

This might be your chance of some significant input to the risk list. Understand the changes that are being implemented. What's being reused, what's the new development, who's the experienced crew, who's new. New and totally uncharted territory vs. known land. That's something you can get from the developers. Not on a silver plate but they might have an idea early in the project about where and what will be impacted. Architects are a similar good source for this kind of input. Although they tend to be a bit further into the universe.
The service organisation - well, they knew where it failed last time, and the time before that. Service delivery managers are usually the best source for historic insight into failures that should have been addressed by the testing effort. If they have kept record of their findings (yes, it could be recorded systematically in a database), they can even quantify and group this for you.
And if you, as the responsible test manager, is not able to get useful input from these critical parts of the project organisation, you should consider your position or be prepared for a "memorable project experience".

Management

This might also be a good place to get some input on risks affecting the testing. Not in terms of tangible risks that can be listed and prioritised, but rather in terms of the managements understanding of which risks that are worth focusing on. For one this will give an indication to the test manager whether there is an alignment between the test organisation and (project) managements understanding of risks. If not - then that is the primary risk to address - and stop testing while priorities are being sorted out.

In fact this is where you start. Get an agreed risk definition for the project which is agreed with the management - be it project management or a project owner or sponsor. Then you know where to aim, and where to prioritise as a test manager.

For the record - this blog is not dead, it just took at 14 week vacation. Happy testing. We're back in action.

>M

QA Engineer walks into a bar...

I came across this one a while back - A fun reminder on equivalence partitioning and test cases you can get from different equivalence classes.

Happy testing !

/Nicolai

Management Potential

I came across an interesting view on the (test) management discipline, during a course I attended this week. The discussion was on what it takes to be a manager, and this was quantified in the following formula:

Management Potential = P * E * L * M

P: Personality

E: Experience

L: Leadership

M: Management

The points that I find interesting about this way of perceiving the manager’s potential is:

Four key elements define the cornerstones of skills that are needed. They are all essential, so having no skill / ability to operate equals a management potential of zero. Good managers need not be masters of all, but can compensate for lack in one area by being skilled in another.

So what is the reasoning behind this discussion? Think about your own management potential, maybe you want to cultivate one area that you are weak in, or maybe you would like to strengthen a skillset to compensate for another? In my opinion, especially inexperienced managers can benefit from putting some thought into how they compensate for lack of experience (while building it).

Food for thought – Have a nice weekend & Happy testing!

/Nicolai

Guiding your estimates in the right direction

Problem: Estimating effort for (test-)activities is inaccurate

Estimation, or educated guessing is the foundation for many an activity related to management of a project. Given that estimation is not an exact science there will be inaccuracy in the numbers used for planning and resource allocation.

 

Solution: Apply general estimation rules and use a technique to guide your estimates.

In order to improve accuracy in your estimation you can rely on some general estimation rules and use a technique to guide your estimation.

General estimation rules:

·         A full-time resource will only be productive around 80% of the time.

·         Shared resources, working on multiple projects will have a lower productivity given that they will have to spend time when they do context switching.

·         There is a high degree of optimism in estimates, as most people have tendency of underestimating complexity and time consumption.

·         Base you estimates on multiple sources – Don’t just use own experience, pursue experience of others if possible.

·         Estimation of a task should be done by the person/team responsibility for doing the task.

·         Remember to include issue management, meetings and other supportive activities in the estimate.

·         Break estimates down if possible.

·         Make sure that assumptions, exceptions and limitations are clearly documented as part of the estimate

Techniques for estimating

·         Top-down estimation

·         Bottom-up estimation

·         Combined top-down & Bottom-up estimation

·         Comparative estimation

·         Parameter estimation

·         One-point estimation

·         3-point estimation

·         …

The point is that there are numerous techniques that allow you to do more or less scientific estimation – You need to choose the one that fits the purpose and tolerance of your project.

Happy testing!

/Nicolai

Supporting your principles of testing

Problem: Your principles of testing cannot stand alone.

Having spend the time defining the principles on which you want you test to run is not enough – It is definitely a good start, but the principles needs to be anchored in the project through activities and action that turns the principle into practice.

Solution: Walk the talk by supporting the principles with action.

One thing is to have a defined test principles, another thing is to support these in practice in the projects. I am going to have a look at some of the supportive actions related to the 7 Principles of Testing from the ISTQB syllabus, as these are commonly accepted.

1) Testing shows presence of defects, but cannot prove that there are no defects. This means that testing needs to be very structured to ensure that testing probes the right areas. Risk based testing can be an approach to testing the right areas, but no matter how you go about this there is two things I would recommend you to inject in the project to address this principle: A strategy that guides your test planning and priorities and expectation management that ensures that the  steering group, project mangers, external stakeholders etc are aware that test will not prove that there are no errors.  

2) Exhaustive testing is impossible and testing everything including all combinations of inputs and preconditions is not possible. Applying different test techniques such as boundary-value analysis, all-pairs testing and state transition tables aim to find the areas most likely to be defective. Use equivalence partitioning as a driver for designing the test to a level of detail that makes sense for the project.

3) Early testing. Remember the principles of verification and verification?  Apply them early and make a plan for when to do it – Some want the V-model to drive early test, others have different approaches, the one that is successful is the one that ensures that testing happens in a timely manner (read: as early as possible). Another thing that facilitates early test is to have a champion in the project from the start – Someone who advocates good quality already from the project planning and onwards.

4) Defect clustering. Does your risk driven approach consider technical complexity / risk? If not, then you might want to consider this in conjunction with some coverage analyses. Another approach is to do experience driven explorative testing, with the purpose of finding candidates for a thorough test – It is a shortcut to finding those troubled areas.

5) Pesticide paradox. Change your approach every now and then – Switch responsibility for functional areas in the test team. Invite new people in, try explorative test, arrange bug hunts, have a bug-off of testers vs. development or business vs. it-team. Point is that you need to refresh yourself now and then, and you might as well put in your plans and have some fun while doing so.

6) Testing is context depending. Base your strategy on the nature of the system and the error tolerance of the receiver. Don’t juse run all test cases just because you can – Run those that are related to the functions delivered.

7) Absence – of – errors fallacy. Invite those users in early, sooner than later, to gather feedback that ensures that you are moving towards ‘fit for purpose’. Involving users requires careful planning, and since you want early involvement this is something that you need to address ASAP in any project.

Again, my point is that you need to have actions that supports the principles, without this, you principles are worthless. Another point is that you need to drive all of the above aided by common sense. Do not invent a wheel unless you have a need for driving somewhere.

Happy testing!

/Nicolai

Risks against test start

Problem: Test is delayed already before it has started

A test may be delayed due to different reasons, causing the start of test to be delayed. Test not starting on planned time is due to events that violates the start criteria for the test activity.

Solution: Monitor and act on risks against your test start criteria.

In order to monitor the risks you need to know what they are – Most common risks against starting a test is related to delays in activities that are on critical path for your test. I have listed the most common problems I see:

• Missing test component or system.
• High error levels in test component or system.
• Testware not complete.
• Missing or wrong testdata.
• Too many (partial) deliveries for test

There are two ways of approaching these risks – reactive and proactive.

Reactive approach is easy, but introduces waste in your development life cycle:

1. Wait until test is delayed by one of the above.
2. Look for rootcause for dealy. 
3. Eliminate rootcause. 
4. Start test at first possible time. 

Proactive is also easy, but requires a higher degrees of advance planning than the reactive approach:

1. Do a risk identification workshop, listing things that will derail your test.
2. List actions that minimize likelihood and impact of identified risks.
3. Adopt actions in your test plans, and follow-up on these.
4. Do a test readiness review in due time before test starts.

To help you on your way you might want to consider the following actions to mitigate the mentioned risks:

• Missing test component or system.
• Likelihood is reduced through continuous integration, automation of deployment / build process, release management of test deliveries. A high degree of automation is preferred in order to ensure timeliness and correctness.
• High error levels in test component or system.
• Test as part of development and cooperation between developers and testers. Focus should be an early test, and on test of critical items that is a must-have in the following test phases. Furthermore a bug triage session on regular basis might help utilize resources on important (from a test perspective) bug-fixing, rather than random bugfixing.
• Testware not complete.
• Test planning is paramount. The plan needs to include all activities and the delivery these leads up to. Follow up on critical path, and make sure that you update the plan as you go along.
• Missing or wrong testdata.
• Just as there is a need for a test strategy you will need a test data management strategy. 
• Too many (partial) deliveries for test
• Release management will get you far – Know what is in the box, and bundle test to match the partial deliveries. Prioritize test cases based on the release content, to ensure that you do not run cases prematurely.

Happy testing!
/Nicolai

Gold plating = Cost injection

Problem: Gold plating of your product is expensive.

Ever experienced that features are added in your product without any adding any real value? If yes then you have probably been gold plating that delivery of yours. Gold plating is done with the best of intentions and most of the time is appreciated by the customer. However, there are many cases where it is not liked and the gold plating is backfires on your product.

Solution: Stick to implementing approved features.

Usually gold plating is introduced either by the project team or by a project manager, at no cost to the customer. Gold plating does not come cheap however! It can increase operation and maintenance costs significantly and will make a dent in quality. The reason is that it raises complexity, introduces features that are not traceable and operates on the outside of the feature approval process – In other words it is a risk that needs to be mitigated.

Although, gold plating sounds good to everyone, it is bad for the project team in the long run. Gold plating increases the development cost, raises the expectation of the customer by inflating the feature per hour / story-point of development. If you do another project for the same customer, he would again expect you to deliver a product with extra features.

Avoiding Gold Plating is important to stay in control of the product, and ensure that the development effort is spend on value-adding features. If you operate in an agile setup you should see gold plating as a violation of the product backlog setup and priorities. Avoiding gold plating requires discipline. It requires the team to ask themselves if they are adding unnecessary or over-engineered functionality into the application.

Start out by discussing gold plating as part of your retrospective, and the set some ground rules like:

• Never allow add any extra function or features to a story without approval.
• As a product managers and sales people should use the product log to add new features – not the sprint log.
• You must establish proper communication lines within the project team, ensuring that approval from customer is achieved.

Then do a little measurement on number of features per iteration vs. number of features in the accepted sprint log. You might have gold plated the solution, but make sure that the result can still be tested, and ultimately accepted by client, otherwise you just might end up paying a very high price for the golden plates.

Happy Testing!

/Nicolai