Wednesday, 11 February 2015

Your very best friend - The Service Delivery Manager

Working with test also means networking. Since most test management is about getting the product tested and shipped according to deadline a natural part of this networking is with the development organisation. Project managers, developers, testers, business partners and other SME's. They provide essential knowledge and resources to the test effort getting things prepared and executed.



This means we have a tendency to forget our most important body of knowledge - the "service organisation". They tend to take over service of the product at go live and then we forget them. Let's correct this mistake in the future.


First and foremost the service delivery manager should be included in the handover of the product prior to go-live. Essential knowledge from the test team must be embedded in the service organisation. Not as a long trivial list of open defects but as story about product quality, workarounds and other necessary information.



Secondly the service delivery manager sits on a treasure trove of knowledge about real user problems and user behaviour as well as technical problems faced by operations. This can be used by the test organisation for a number of tasks including estimation, prioritisation of test effort, general planning of test and as a source for planning test of "off-normal" situations.

The service delivery manager is as such an essential member of the test team. Not on a permament basis but ad hoc at the right time during the project the Service Delivery Manager is your best friend.
If the service delivery manager is not able to provide the input, he is at least the gateway to the organisation where the knowledge is present and available. Just kick in the door.

Monday, 9 February 2015

TDD – My experience

Reading this excellent post at ’The Codeless Code’ I came to think of my own experience with Test Driven Development (TDD): http://thecodelesscode.com/case/44
 
My first take on Test Driven Development was many years ago, before agile had really made an impact. We faced the following challenges:
  • Customer wanted shorter development cycles.
  • Development was outsourced.
  • Test resources was scarce and had no coding skills.
  • Offshore resources had little business domain knowledge.
 
In order to address especially the last two bullets it became evident that we had to change strategy, and focus on implementing a process that supported the offshore development team and enabled the onshore resources to assist the offshore team though reviews and guidance.
 
TDD was introduced as part of the low level specification done by the offshore team. One of the sections in the specification was dealing with the unit test cases that had to be written in order to cover the functionality detailed in the spec.
 
Another challenge was that the offshoring happened fast, and we had little time to train the team. That meant that we had to come up with ways of “testing” the team’s understanding before countless hours was used coding a solution. We found that testing the solution late in the development cycle often proved too late to counter misunderstandings. This meant that we came up with this very simple procedure for handing over the development assignment to the offshore team:
 
Walkthrough of business requirements and related high level specification, to empower the offshore team to take up development. Offshore team did low-level detailed specifications on solution including test cases, and these was sent for review. Testers reviewed test cases, architects reviewed the solution, and sr. developers reviewed the pseudo code for completeness and capability with current solution. The interesting thing was that it almost always proved to be the test cases that gave the pointers on whether or not the proposed solution was in line with expectations.
 
Once low-level design was approved the test cases was implemented – Once done they were checked in and added to system codebase, and baselined as part of the delivery. After development of the actual solution all test cases was part of the regression test suite, meaning that we soon had lots of automated test on the project leaving us with a high level of code coverage.
 
The real challenge of introducing TDD was to shape the organization to facilitate this new way of working, and enforcing the procedures. There was quite a stir in the onshore organization, not only did they have to embrace the new offshore colleagues, they also had to hand over some of their assignments to them. At first the offshore colleagues were put off by the constant review and scrutiny of their code – Little code could be written before passing a series of quality gates. These gates was not in any of the standard development processes, meaning that they were sailing uncharted waters. This meant that there was a lot of explaining and discussion up front before the TDD approach could be attempted.
 
Biggest impact was on the testers – they had to abandon trivial functional testing, as this responsibility now rested on the shoulders of the developers. This was hard, as they were used to doing test cases one to one on the functional requirements. Their scope was now expanding to a compiling test results of TDD into test coverage reporting, and then doing testing in areas that looked a little weak. On top of this, they were now in charge of the factory acceptance, calling for testing that was focused on the system as a whole, and challenging their business domain knowledge far more than they were used to.
 
Happy testing!
 
/Nicolai
 
PS. MSDN has a very nice guide for those wanting to pursue TDD:
http://msdn.microsoft.com/en-us/library/aa730844(v=vs.80).aspx
 

Tuesday, 3 February 2015

Enforcing your quality standards


Problem: Enforcing the quality standards.

You spend months negotiating the quality policy and test strategy for your development department, and now it is time to implement, but nothing is happening. The quality of your products remain the same and there is little to no measurable change that points towards better quality.

Solution: Identify quality rules and enforce them as part of your build process.

Assuming that you have your quality goals in a quality policy and/or a test strategy you will be looking for means to inject it in the development organization. Focus on a few goals at a time, seeking fact-based follow-up on tends, and clear goals for how to enforce that the means to reach the goals.

A word of advice – Without management commitment you will not get very far with your efforts, so a good place to start is to get management buy-in on enforcing that quality policy. Another thing to think about is that you need help from all members of your development community, meaning that you will have to bring arguments that can be understand across the organization when explaining the changes. Avoid communication like this:


A place to start is the check in rules / committing code. Here you have a chance to enforce a quality gate based on rules that support the goals you have. Furthermore it is important to ensure that the state of current build is visible to all who works on it, allowing quick response on problems. Collect metrics that allow your monitoring on trends, and use these metrics to support the quality policy goals.

Communicate your goals, and how they are measured. Eg. Test coverage must be equal to or higher than last build – Measured as line coverage delta in build report vs. previous build. Define build break rules around these measurements, and stop builds that violate / jeopardize quality.

Another thing that can support the check in rules is to outline best practice. Simple guidelines will get you far and promote a higher standard of the builds. Formulate these guidelines, with statements like: Don’t check in on a broken build. Always run commit test locally before committing. Do not comment out failing tests.

Finally you can consider to do personalized measurements from the trends gathered on quality metrics, and have that as a driver for one’s ability to get a bonus or raise.

Happy testing!

/Nicolai


PS. Just came across this blog, promoting toolbased build monitoring:
http://www.federicosilva.net/2015/02/about-continuous-integration-and-build.html

Thursday, 29 January 2015

Following up on the project risk log

Problem: Measuring progress to facilitate risk management.
Many a risk-log has ended it's life without any real value being added, due to lack of implementation and follow-up. The reason is, in my experience that the project fails to use test metrics for follow-up and governance.
Solution: Baseline the risk register on regular bases and compare metrics.
In order to baseline the risk log you would require a document containing the risks and some metrics. There are many ways of working with risks, but most common is the probability and consequence. This allows the project to plot the risks in a risk matrix, detailing the priority that the risks should have when working with mitigation. The matrix might look like this:
 
From here on the fun starts! Based on the risk mitigation activities that you implement you should see changes in the risk matrix. E.g. if your strategy is to address high probability risks first, you should see a trend of risks moving downwards in the matrix, and here is how it works:
On a monthly basis, you compare the risk matrix to last month, and you look for the following:
  • No change = Your risk management process is not kicking in, as the log is not changing!
  • Fewer risks = Your risk management is either mitigating those risks, or they are turning into issues.
  • More risks = Project is getting wiser, risks are identified – Food for thought and planning / mitigation.
  • Risks are migrating South & West = You are mitigating the probability and Impact of the risks.
  • Risks are migrating North & East = Your efforts are not mitigating the probability and Impact of the risks.
By doing this you will raise awareness on the risk management exercise, and force stakeholders to engage in a dialogue on risk within the project.
Another and even simpler way of baselining is to do a count of the number of risks and their status (assuming that you operate with one). You might have a list like this:
Potential: 14
Actual: 25
Averted: 12
Irrelevant: 3
Total: 54
Comparing the numbers on a regular basis will tell you if your risk management process is moving the risks in the right direction. You can even do a little trend-graph to keep things manager-friendly.
Finally, the use of tools will make it much easier for you in the long run – I’m not going to promote any tools, as I have no preference. One word of advice is to use Dr. Google and look for a risk management template for Excel, that should get you quite far for free. Look for a template draws the risk matrix for you.
Happy testing!
/Nicolai
 

Friday, 23 January 2015

Static test planning


Problem: Planning the verification part of testing does not always happen in a structured way.

Reviewing aka static testing is the verification part of the V&V used for checking a delivery. Documentation is very important to testers, and testing artifacts and test basics are in need of processes to support them in their lifetime. A part of this is reviews and the approval of the document that it leads to – Something that require careful planning, as it requires that timing and resources are aligned to be done in due time.

Solution: Static test planning

Base your static test om a plan, much like you would do in the case of dynamic testing. The plan should be build on same principles as you dynamic test plan. Following items should (as a minimum) be detailed in your static test plan:

  • Deliverables under review – Look at the list of deliveries and nominate those that needs review.
  • Approach – Describe the methods/types of review applied for each document under review.
  • Document approval – What are the prerequisites for approval of a document?
  • Procedures and templates – What is the procedure for reviewing and what tools are needed?
  • Responsibilities – Who is document owner, review lead and approver?
  • Staffing and training needs – Who will do the work? Do they need training I doing reviews?
  • Schedule – When should the work be done? Remember your dependencies to project plan.

What is the purpose of static testing? Much like dynamic testing, we will be looking for deviation from standards, missing requirements, design defects, non-maintainable code and inconsistent specifications. The process of review also resembles the one for dynamic test:

Define expectations to review, then perform review & record results and finally implement changes required. When the review is done and findings have been addressed then look at sign off of the document.

Happy testing!

/Nicolai

Friday, 9 January 2015

The quality of testing


Just recently I attended a venue, where the quality of testing was the topic of the day. There was some interesting points about how this could be measured and monitored that I would like to share.

The discussion had the following topics all related to quality of the testing:

  • Completeness of the test done
  • Credibility of the test documentation
  • Reliability
  • Quality Parameters to be observed

The discussion was revolving around the parameters that was driving quality of a product, and especially the quality found in the work done by testers. It quickly became evident that the work and quality of it is hard to quantify, and require that you look into the details of the work, looking for completeness, creditability and reliability.

You can see the parameters we discussed below.

Completeness of the test done:

  • Coverage vs. objects under test
    • Actual coverage vs. Planned
    • Execution vs. High risk areas first and coverage of same.

Creditability of the test effort

  • Test cases are created and executed based on objective foundation
    • Structure in test cases.
    • Independence in review and test.
    • Dedicated resources for testing.
    • Separate estimate / time for test activities.

Reliability in test

  • Predictability in test result (based on metrics and pervious experience).
  • Status on test can easily and accurately be derived at any time.
  • Defects and other observations are handled according to defined process.

Quality Parameters to be observed

  • Defined metrics, that are monitored (less is more)
  • Approval criterias and compliance in the deliveries (eg. Open defects vs. categories)
  • Test efficiency per test phase (& number of defects found after go-live)

…And there are probably a lot more if you really want to scrutinize that test department of yours – Just remember to have a little faith in the work done, before going all in.

Happy testing & Have a nice weekend!

/Nicolai

Wednesday, 24 December 2014

Happy Holidays!


Hi all,

Another year passed, and I would like to wish you all the best & happy holidays.

 

Thanks for all the positive feedback received this year!

/Nicolai