Problem:
All defects are critical just before launch – defect priority can easily loose
it’s value & meaning
Clock is
ticking and a release is approaching. Development only have a few days/hours
before code-freeze and only the most urgent bugs can be addressed. This is
where defect priority can loose its meaning, as people see all bugs as
critical, hoping for them to make the cut.
Solution: STOP!
Check your defect metrics and call for a bug triage
Most
extreme example I have seen was a large program where the % of priority 1 &
2 defects went from 17% to 92% in a week. This is what happened:
Program management
announced to all stakeholders that deadline was tight, and that time would no
longer pay attention to defects of severity 3 or lower. In the two days that
followed this announcement, we could see a pattern forming. More than half of
the existing defects had priority changed, majority got priority 1 or 2. 90+%
of new defects was opened with priority 2+.
This story
holds two lessons:
- Do not tell stakeholders that you will be ignoring their defects – They will not take it kindly.
- Make sure to check your defect metrics from time to time and do a bug triage when needed.
Since this
I have always kept a pie-chart of defect priority (and severity) at hand in my projects –
This is easy, as you get it for free in most defects tracking tools. Checking it
from time to time, will offer a sanity check of priority spread, before doing a
bug triage exercise.
In the
example above priority got confused with severity, everyone got all jumpy about
when, and if things got fixed. This meant that the fist bug triage meeting was
a battle of will – Who was prepared to give ground and sacrifice some bugs to
lower priorities? The meeting did not change any priorities, and we simply
agreed that all stakeholders (business areas) had to nominate 3 bugs that they
needed and 5 they wanted, and that set the new standards for priorities – Not very
pretty but it did the job, giving development priorities for completing bug
fixing in the time left.
For more
information on bug triage have a look at this: http://www.softwaretestingtimes.com/2010/04/bug-triage-meeting-process.
Have a nice
weekend & Happy testing!
/Nicolai
Hi Nicolai
ReplyDeleteGreat article! I think the solution you have suggested is the right way to go.
I say this based on my experience in one of the projects I handled as the Test Manager.
Initially the project I was working on, got two lifelines and was postponed by almost a month but then came one final cut when the top management refused any further extensions. As a result the Program Director gave strict instructions to the entire team to focus only on Blocker and Critical defects. Major and Minor defects were to be taken up in a subsequent patch release.
Within 2-3 days I started seeing a similar pattern as you mentioned, where most of the defects were being logged with Critical severity and that too this was coming from 2 of the team members. One day I had to talk to them and ask, as to what was going on and how come they were finding the critical defects more than the others, who were more senior to them both in terms of overall experience as well as in terms of application knowledge. I was shocked to hear the reason provided. They were logging everything as Critical because the Program Director had asked Developers to focus only on Blockers and Critical ones. They wanted to make sure their defects were fixed before the go-live. Hence they logged everything as Critical so that it got fixed before the release so that the end user would not face that defect.
I had to sit with them and explain that while their intentions were good, their thought process was not correct. After that, I sat with the Project Manager and Dev Lead to triage all the critical defects logged in the last 1 week and re-visited their severity. It turned out almost 90% of the defects logged by those 2 team members had incorrect severity given. We were able to significantly bring down the number of defects to work upon and were able to manage the release on time.
I was able to catch the Defect severity spread quickly, thanks to the metrics we were preparing, one of which was a pivot table of Team Members v/s. Defect Severity. Another useful metric to monitor is Component or Module wise v/s. Defect Severity.
In my opinion, both these are very useful metrics to monitor, at any stage of the project, especially the latter one.
- Hemal
Hi Hemal,
DeleteThanks for your comment, good point about the sit-down – Bug triages need not be a very formalized event, where huge process machines are fired up and lots of people are involved. A sit-down like the one you describe is the pragmatic version of a bug triage meeting, and I am all for the pragmatic solution.
Furthermore looking for users / user groups who contribute with most critical bugs is a nice angle, as that allows the test & project managers to target communication to those most in need of direction.
Have a nice day!
/Nicolai