Performance testing - a real world experience and day-zero considerations

Welcome to 2016. Why not kick off the new year with an old story. Some years back I was a test manager for a large project aiming at launching a business critical system. The test strategy recommended a few structured performance test activities aiming at proving that the system would actually be able to deal with the expected number of users, that peak log on (during the mornings) could be handled and that system resources would be freed up as users log off.

All of these recommendations were approved by project management and a separate team was set up to design, implement and execute the necessary tests. This would be done late in the project phase and was as such totally normal. Do the testing at a point where the software is sufficiently mature to actually be able to trust and use the test results for a go/no-go decision. So far, so good.

Since this was a project that would implement a solution that would replace an existing solution we didn't have to guess too much on the user behaviour for normal use. Just look in the log files and find patterns. A large part of the testing was designed around this knowledge.

Then we consulted the implementation team to figure out how they expected to roll out the solution to the organisation. We returned with the knowledge of a "big bang" implementation. There were no alternatives so we also needed this as a scenario. How would the solution scale and behave on the first day when everybody had an email in their inbox saying "please log on to this brand new and super good system"?

No problems so far. Knowing that the organisation was located in two different time zones that took some of the expected peak load off and we didn't have to have this cruel "100% users at the same time"-scenario. Emails to different parts of the organisation could be sent out to groups of users with say 10-15 minutes intervals to avoid a tidal wave of concurrent log ons. Good and pragmatic idea and that was agreed in the project and executed by the implementation team.



The one thing we didn't take into account was how organisations and especially middle management works. Middle management tend to send a lot of mails around these days. In ways not always known or controlled by a project like ours. So in the real world we succeeded with our performance testing but failed on day-zero.

As soon as middle management started to get the "Important information- log on to this new system" they did what they always do with this kind of information - passed it on. Not only to their own organisation but across the organisation. using different mail groups that would hit 30, 50 or 100 persons at a time. They were used to this in their daily operational life, and to them this was just another operational morning.

The result was that the peaks of log ons were completely different from what we had expected and planned - and tested. Not to the extent that there was a complete meltdown but there was short outages during the first couple of hours - and of course some angry and concerned users who needed feedback and assurance that they could trust the system which was mission critical for them.

Lesson learned: Think a bit outside the box. Not always worst case scenario, but closer than you might think. Even though you have a lot knowledge to build on always consider performance testing for day-zero scenarios as something truly special. First impressions last, especially for real-life users.

Season greetings - and a few thoughts on the testing challenges for 2016

It's that time of year. Many of us are engaged in testing just how much the Christmas mood can be stretched. It's kind of a stress test - this year no exception. We do not do predictions here on this blog. At least not the bold ones. For that we have the experts who every year publish the World Quality Report. If you have a spare moment before Christmas take a look at it. At least it's a good source of inspiration for the strategic initiatives

But back to this blog. What's happening here? Well for 2016 we would like to see the following.

Make your black box testing a bit more white

This is a tough one. One on side black box testing has a specific purpose - interact with a system with known inputs in order to get predicted test results to prove that requirements are implemented. On the other hand - wouldn't it be nice if there actually was a deeper understanding among the testers about what actually goes on behind the scenes? Maybe that's also what is covered in the World Quality Report when they put emphasis on "Expand testing teams’ skills beyond manual and test automation".

Spend more resources on test data

Test data will continue to become more and more critical for testing. For system integration testing it is maybe the most important component to be in control of. And yet, this is where most testing organisations fail to address the problems they are facing. This is not about strategy or big frameworks. This is hands on work that require the participation of the development organisation on a broad scale - not the same as many resources or a lot of effort but do a bit of agile approach to this and focus on the areas with high frequency and where it really hurts. It's like a toothache - it does not go away on itself and every time you test the problem is there - and it takes more resources to work with a bad solution so no need for fancy ROI analysis. Just do it.

Devops and testing are two of a kind

If your organisation is beyond "agile" and are now focusing on the new management buzz "Devops" make sure that testing is involved in whatever activities that happen around this area. We all know that testers are the most valuable source of knowledge prior to go-live and the "Ops" should therefore be interested in that knowledge. And vice-versa - testers always love real "Ops" stories because they give such good input to fill the missing blanks. The areas we forgot to think about when we did our test planning and analysis. But the end users found them - those who pay our salary.

Season greetings

So please, Santa. This was three of the wishes on the list. There are about 999 other interesting test related topics that we would also like to address but they are currently in the drawer waiting for 2016 and more action on this blog.

Season greetings to all of you and thanks for following this blog.

Black box testing - FAIL

Unless you've just arrived to planet Earth from some other galaxy you cannot have missed the biggest news story in software this year - and perhaps also this decade. VW's "cheat software" that made the emission tests from various diesel engine models behave differently when they went through the US EPA emission tests - only to behave differently once they were driven by the right foot of real people.

No matter who decided what, who knew what - and who did what -the implications from a software testing perspective are interesting. First of all this is a signal to everybody about not trusting what's inside "the box". If you work professionally with test, software audit or other types of certification jobs where software is included the black box testing activities must undergo increased scrutiny.

A much more focused approach to the vendors black box testing activities should be the result of this event and a much better understanding of the tests and results should be the outcome. It is of course not possible to have completely transparent white box testing activities all the way through the development process and even it was the feasibility of looking over the shoulder is not really an option.

One simple learning is to limit the re-use of test cases. Make sure that the test is flexible within certain boundaries. Secondly do not rely on tests that are known by test vendor - and definitely not too much on test cases from their black box testing activities.

Instead, when you are involved in FAT and SAT activities your main goal should be to understand the testing scope of your vendor and challenge that with your own black box test cases. A further approach should be to have a team of skilled testers and end users involved in exploratory testing - a set of planned and focused activities with the vendors software prior to accepting delivery.

VW is presumably not the only company where the content of the box is different than expected and the motivation for spending extra time and resources is now forever embedded in their specific case. The ROI from a customer perspective is quite easy to calculate - insufficient testing = don't know how the engine of your business is running.

From a historical perspective software has often included unwanted or hidden features. Just read a few examples about Office 97.

PRINCE2 quality management strategy

I have spend the past months working in an organization that utilizes PRINCE2 as driver for the projects run. One of the products I have been working with is the Quality Management Strategy (QMS), an essential work product that defines how to approach quality in a project.

The QMS describes how the quality management systems of the participating organizations will be applied to the project and confirms any quality standards, procedures, techniques and tools that will be used. Sounds easy enough, but requires a little thought to ensure proper level of information to ensure that quality is manageable in the project.

The QMS is created as part of the project initiation, and continues to be updates throughout the project. It contains the following:

Quality management procedures for planning and control. Tools and techniques, including records and reporting. Quality management activities, covering dependencies, roles and responsibilities.

During my short time working with the QMS work product, I have made the following conclusions:

• Gold-plating your QM will cost you a lot – Make sure that you set your target at fit project purpose and scope. Make sure that scale and tailoring of standards is described. You might not need everything related to quality that you can find on the corporate intranet. Less is more – too little is a problem.

• Project quality is about customer happiness and satisfaction, meaning that you need a lead on those acceptance criterias and definitions that tells if product is fit for purpose. Furthermore you need to think about how to get input from the customers.

• Remember that PRINCE2 is about learning, meaning that you should have some focus on improving described processes and tools, based on the observations and learning you experience in your project. These activities needs a place in QMS.

Happy testing!

/Nicolai

A few thoughts about risk

We've been around this topic a number of times - risk based testing. Another Danish blogger did a short post about this (in Danish) just before the summer holidays.

The point is right - base the test approach on the identified risks. There are numerous problems in that approach. A few mentioned below.

Customer input

The "stakeholders" who should be able to list and prioritise risks hardly know what they've bought or ordered. Most organisations or customers have little idea about what's going to hit them when they embark on projects where risk based testing could be of good use. They are about to participate in the expectation roller coaster ride where each day will bring new challenges and decisions forward and on top of that they have to prepare their own organisation and their customers for the changes they've ordered. They are not used to working with the fluffy term of "risks" and being the test manager trying to facilitate them is no easy task.

The professional testers

This group of project participants have to be kept on a short leash when it comes to risks. Depending on their interest, experience and beliefs they can carpet bomb any risk session in a matter of minutes. Most of them with very relevant input, but too often it is input that is way outside what the project is capable of dealing with - even with lots of resources (time, hands and money).
That said, testers are usually your best chance of success when it comes to qualified risk input based on their experience but put on your angry test manager hat and cut away any risks that are improbable or impossible to deal with.

Development and service organisation

This might be your chance of some significant input to the risk list. Understand the changes that are being implemented. What's being reused, what's the new development, who's the experienced crew, who's new. New and totally uncharted territory vs. known land. That's something you can get from the developers. Not on a silver plate but they might have an idea early in the project about where and what will be impacted. Architects are a similar good source for this kind of input. Although they tend to be a bit further into the universe.
The service organisation - well, they knew where it failed last time, and the time before that. Service delivery managers are usually the best source for historic insight into failures that should have been addressed by the testing effort. If they have kept record of their findings (yes, it could be recorded systematically in a database), they can even quantify and group this for you.
And if you, as the responsible test manager, is not able to get useful input from these critical parts of the project organisation, you should consider your position or be prepared for a "memorable project experience".

Management

This might also be a good place to get some input on risks affecting the testing. Not in terms of tangible risks that can be listed and prioritised, but rather in terms of the managements understanding of which risks that are worth focusing on. For one this will give an indication to the test manager whether there is an alignment between the test organisation and (project) managements understanding of risks. If not - then that is the primary risk to address - and stop testing while priorities are being sorted out.

In fact this is where you start. Get an agreed risk definition for the project which is agreed with the management - be it project management or a project owner or sponsor. Then you know where to aim, and where to prioritise as a test manager.

For the record - this blog is not dead, it just took at 14 week vacation. Happy testing. We're back in action.

>M

QA Engineer walks into a bar...

I came across this one a while back - A fun reminder on equivalence partitioning and test cases you can get from different equivalence classes.

Happy testing !

/Nicolai

Management Potential

I came across an interesting view on the (test) management discipline, during a course I attended this week. The discussion was on what it takes to be a manager, and this was quantified in the following formula:

Management Potential = P * E * L * M

P: Personality

E: Experience

L: Leadership

M: Management

The points that I find interesting about this way of perceiving the manager’s potential is:

Four key elements define the cornerstones of skills that are needed. They are all essential, so having no skill / ability to operate equals a management potential of zero. Good managers need not be masters of all, but can compensate for lack in one area by being skilled in another.

So what is the reasoning behind this discussion? Think about your own management potential, maybe you want to cultivate one area that you are weak in, or maybe you would like to strengthen a skillset to compensate for another? In my opinion, especially inexperienced managers can benefit from putting some thought into how they compensate for lack of experience (while building it).

Food for thought – Have a nice weekend & Happy testing!

/Nicolai