Season greetings - and a few thoughts on the testing challenges for 2016

It's that time of year. Many of us are engaged in testing just how much the Christmas mood can be stretched. It's kind of a stress test - this year no exception. We do not do predictions here on this blog. At least not the bold ones. For that we have the experts who every year publish the World Quality Report. If you have a spare moment before Christmas take a look at it. At least it's a good source of inspiration for the strategic initiatives

But back to this blog. What's happening here? Well for 2016 we would like to see the following.

Make your black box testing a bit more white

This is a tough one. One on side black box testing has a specific purpose - interact with a system with known inputs in order to get predicted test results to prove that requirements are implemented. On the other hand - wouldn't it be nice if there actually was a deeper understanding among the testers about what actually goes on behind the scenes? Maybe that's also what is covered in the World Quality Report when they put emphasis on "Expand testing teams’ skills beyond manual and test automation".

Spend more resources on test data

Test data will continue to become more and more critical for testing. For system integration testing it is maybe the most important component to be in control of. And yet, this is where most testing organisations fail to address the problems they are facing. This is not about strategy or big frameworks. This is hands on work that require the participation of the development organisation on a broad scale - not the same as many resources or a lot of effort but do a bit of agile approach to this and focus on the areas with high frequency and where it really hurts. It's like a toothache - it does not go away on itself and every time you test the problem is there - and it takes more resources to work with a bad solution so no need for fancy ROI analysis. Just do it.

Devops and testing are two of a kind

If your organisation is beyond "agile" and are now focusing on the new management buzz "Devops" make sure that testing is involved in whatever activities that happen around this area. We all know that testers are the most valuable source of knowledge prior to go-live and the "Ops" should therefore be interested in that knowledge. And vice-versa - testers always love real "Ops" stories because they give such good input to fill the missing blanks. The areas we forgot to think about when we did our test planning and analysis. But the end users found them - those who pay our salary.

Season greetings

So please, Santa. This was three of the wishes on the list. There are about 999 other interesting test related topics that we would also like to address but they are currently in the drawer waiting for 2016 and more action on this blog.

Season greetings to all of you and thanks for following this blog.

Black box testing - FAIL

Unless you've just arrived to planet Earth from some other galaxy you cannot have missed the biggest news story in software this year - and perhaps also this decade. VW's "cheat software" that made the emission tests from various diesel engine models behave differently when they went through the US EPA emission tests - only to behave differently once they were driven by the right foot of real people.

No matter who decided what, who knew what - and who did what -the implications from a software testing perspective are interesting. First of all this is a signal to everybody about not trusting what's inside "the box". If you work professionally with test, software audit or other types of certification jobs where software is included the black box testing activities must undergo increased scrutiny.

A much more focused approach to the vendors black box testing activities should be the result of this event and a much better understanding of the tests and results should be the outcome. It is of course not possible to have completely transparent white box testing activities all the way through the development process and even it was the feasibility of looking over the shoulder is not really an option.

One simple learning is to limit the re-use of test cases. Make sure that the test is flexible within certain boundaries. Secondly do not rely on tests that are known by test vendor - and definitely not too much on test cases from their black box testing activities.

Instead, when you are involved in FAT and SAT activities your main goal should be to understand the testing scope of your vendor and challenge that with your own black box test cases. A further approach should be to have a team of skilled testers and end users involved in exploratory testing - a set of planned and focused activities with the vendors software prior to accepting delivery.

VW is presumably not the only company where the content of the box is different than expected and the motivation for spending extra time and resources is now forever embedded in their specific case. The ROI from a customer perspective is quite easy to calculate - insufficient testing = don't know how the engine of your business is running.

From a historical perspective software has often included unwanted or hidden features. Just read a few examples about Office 97.

PRINCE2 quality management strategy

I have spend the past months working in an organization that utilizes PRINCE2 as driver for the projects run. One of the products I have been working with is the Quality Management Strategy (QMS), an essential work product that defines how to approach quality in a project.

The QMS describes how the quality management systems of the participating organizations will be applied to the project and confirms any quality standards, procedures, techniques and tools that will be used. Sounds easy enough, but requires a little thought to ensure proper level of information to ensure that quality is manageable in the project.

The QMS is created as part of the project initiation, and continues to be updates throughout the project. It contains the following:

Quality management procedures for planning and control. Tools and techniques, including records and reporting. Quality management activities, covering dependencies, roles and responsibilities.

During my short time working with the QMS work product, I have made the following conclusions:

• Gold-plating your QM will cost you a lot – Make sure that you set your target at fit project purpose and scope. Make sure that scale and tailoring of standards is described. You might not need everything related to quality that you can find on the corporate intranet. Less is more – too little is a problem.

• Project quality is about customer happiness and satisfaction, meaning that you need a lead on those acceptance criterias and definitions that tells if product is fit for purpose. Furthermore you need to think about how to get input from the customers.

• Remember that PRINCE2 is about learning, meaning that you should have some focus on improving described processes and tools, based on the observations and learning you experience in your project. These activities needs a place in QMS.

Happy testing!

/Nicolai

A few thoughts about risk

We've been around this topic a number of times - risk based testing. Another Danish blogger did a short post about this (in Danish) just before the summer holidays.

The point is right - base the test approach on the identified risks. There are numerous problems in that approach. A few mentioned below.

Customer input

The "stakeholders" who should be able to list and prioritise risks hardly know what they've bought or ordered. Most organisations or customers have little idea about what's going to hit them when they embark on projects where risk based testing could be of good use. They are about to participate in the expectation roller coaster ride where each day will bring new challenges and decisions forward and on top of that they have to prepare their own organisation and their customers for the changes they've ordered. They are not used to working with the fluffy term of "risks" and being the test manager trying to facilitate them is no easy task.

The professional testers

This group of project participants have to be kept on a short leash when it comes to risks. Depending on their interest, experience and beliefs they can carpet bomb any risk session in a matter of minutes. Most of them with very relevant input, but too often it is input that is way outside what the project is capable of dealing with - even with lots of resources (time, hands and money).
That said, testers are usually your best chance of success when it comes to qualified risk input based on their experience but put on your angry test manager hat and cut away any risks that are improbable or impossible to deal with.

Development and service organisation

This might be your chance of some significant input to the risk list. Understand the changes that are being implemented. What's being reused, what's the new development, who's the experienced crew, who's new. New and totally uncharted territory vs. known land. That's something you can get from the developers. Not on a silver plate but they might have an idea early in the project about where and what will be impacted. Architects are a similar good source for this kind of input. Although they tend to be a bit further into the universe.
The service organisation - well, they knew where it failed last time, and the time before that. Service delivery managers are usually the best source for historic insight into failures that should have been addressed by the testing effort. If they have kept record of their findings (yes, it could be recorded systematically in a database), they can even quantify and group this for you.
And if you, as the responsible test manager, is not able to get useful input from these critical parts of the project organisation, you should consider your position or be prepared for a "memorable project experience".

Management

This might also be a good place to get some input on risks affecting the testing. Not in terms of tangible risks that can be listed and prioritised, but rather in terms of the managements understanding of which risks that are worth focusing on. For one this will give an indication to the test manager whether there is an alignment between the test organisation and (project) managements understanding of risks. If not - then that is the primary risk to address - and stop testing while priorities are being sorted out.

In fact this is where you start. Get an agreed risk definition for the project which is agreed with the management - be it project management or a project owner or sponsor. Then you know where to aim, and where to prioritise as a test manager.

For the record - this blog is not dead, it just took at 14 week vacation. Happy testing. We're back in action.

>M

QA Engineer walks into a bar...

I came across this one a while back - A fun reminder on equivalence partitioning and test cases you can get from different equivalence classes.

Happy testing !

/Nicolai

Management Potential

I came across an interesting view on the (test) management discipline, during a course I attended this week. The discussion was on what it takes to be a manager, and this was quantified in the following formula:

Management Potential = P * E * L * M

P: Personality

E: Experience

L: Leadership

M: Management

The points that I find interesting about this way of perceiving the manager’s potential is:

Four key elements define the cornerstones of skills that are needed. They are all essential, so having no skill / ability to operate equals a management potential of zero. Good managers need not be masters of all, but can compensate for lack in one area by being skilled in another.

So what is the reasoning behind this discussion? Think about your own management potential, maybe you want to cultivate one area that you are weak in, or maybe you would like to strengthen a skillset to compensate for another? In my opinion, especially inexperienced managers can benefit from putting some thought into how they compensate for lack of experience (while building it).

Food for thought – Have a nice weekend & Happy testing!

/Nicolai

Guiding your estimates in the right direction

Problem: Estimating effort for (test-)activities is inaccurate

Estimation, or educated guessing is the foundation for many an activity related to management of a project. Given that estimation is not an exact science there will be inaccuracy in the numbers used for planning and resource allocation.

 

Solution: Apply general estimation rules and use a technique to guide your estimates.

In order to improve accuracy in your estimation you can rely on some general estimation rules and use a technique to guide your estimation.

General estimation rules:

·         A full-time resource will only be productive around 80% of the time.

·         Shared resources, working on multiple projects will have a lower productivity given that they will have to spend time when they do context switching.

·         There is a high degree of optimism in estimates, as most people have tendency of underestimating complexity and time consumption.

·         Base you estimates on multiple sources – Don’t just use own experience, pursue experience of others if possible.

·         Estimation of a task should be done by the person/team responsibility for doing the task.

·         Remember to include issue management, meetings and other supportive activities in the estimate.

·         Break estimates down if possible.

·         Make sure that assumptions, exceptions and limitations are clearly documented as part of the estimate

Techniques for estimating

·         Top-down estimation

·         Bottom-up estimation

·         Combined top-down & Bottom-up estimation

·         Comparative estimation

·         Parameter estimation

·         One-point estimation

·         3-point estimation

·         …

The point is that there are numerous techniques that allow you to do more or less scientific estimation – You need to choose the one that fits the purpose and tolerance of your project.

Happy testing!

/Nicolai