Handling Non-Required Data in Bug Reports – Part 1
Posted by Eric Jacobson at Tuesday, November 27, 2012Congratulations, you just found a bug in an office supply commerce application! It appears that any time you submit an order with more than one line item, an “object reference not found” error is thrown. Cool! Here are the repro steps you logged in the bug report:
- Create a new order.
- Add Pencils to the new order.
- Add Pens to the new order.
- Select UPS for shipping method.
- Select Credit Card for payment method.
- Submit the new order.
Expected Results: No errors are thrown.
Actual Results: “Object reference not found” error is thrown.
Those are pretty solid repro steps. Every time one performs them, the Actual Results occur. Nevertheless, do you see any problems with the above repro steps?
Hmmmmm….
What if you depend on just the repro steps to understand the bug?
Does this bug only repro if pens and pencils are ordered, if the UPS shipping method is selected, if the Credit Card payment method is selected?
Those details capture what the tester did, but might they mislead? Anytime you add specific data to your repro steps, you may be implying, to someone, that said data is required to repro the bug (e.g., perhaps the “pens” data in our DB is bad). A more skilled tester may log the bug like this:
- Create a new order.
- Add at least two line items to the new order.
- Select a shipping method.
- Select a payment method.
- Submit the new order.
Okay, that seems considerably better to me. However, we can still improve it. Is it reasonable to assume the programmers, testers, and business analysts on our project team know how to submit an order? Do they already understand that in order to submit an order, one must specify shipping and payment methods? Yes!
Thus, an even more skilled tester may end up with these repro steps:
- Create a new order.
- Add at least two line items to the new order.
- Submit the new order.
There is nothing extra to cloud our interpretation. The steps look so minimal and easy, anyone could do them! Now that’s a set of repro steps we can be proud of.
RSS
Sorry, but I don't agree at all. I'd much rather have the first of these reports.
Have you tried every combination (perhaps permutation) of line items? And mixed those with shipping and payment methods? If not, you're just guessing. And when I fail to reproduce the bug I have to come to you and ask you what you actually did to cause the problem.
I'll happily take extra information (I also tried with these items; it failed with every shipping method I tried; I couldn't find any input with more than one item which didn't cause the problem; my guess is ...) but please give me something concrete rather than your extrapolations.
Not really :)...from my experience with BA,PM,Devs,Managers....they usually ask 'is this happening for other payment methods' ?
I'm not sure they have all required information if you go with your suggested format. In the end, the bug report is for everyone to better understand the circumstances of the failure, it's not meant to be time efficient only.
Isn´t the purpose of a report to give useful information to understand the bug? I think every bug report has to answer some questions that anyone who read it can solve. For example: What is happening? Where is happening? How critical is this problem? How can i reproduce the problem?
Taking the example:
- Create a new order.
**What type of order is this?
**Is there only one kind of order? **If there are many types, Does the problem apply to any order?
- Add at least two line items to the new order.
**Does two mean two of the same item or two different items?
**They have to be two?
- Submit the new order.
**Are the actual results and expected results in the previous description part of this report too?
**How critical is this bug?
I actually see your first description in the blog entry as useful information:
"It appears that any time you submit an order with more than one line item, an “object reference not found” error is thrown."
Hey guys, thanks for the comments.
You seem to think more information is always better. IMO, it's the job of the tester to figure out which information is relevant and which is not.
Logging a bug full of extra information is easy. We can always imagine that other details could play a part in the bug. Maybe the login account is significant. Maybe step one of the repro steps should be:
1. Log in to the application as Eric Jacobson.
A skilled tester should determine the minimum relevant data. In our example, the tester can use the Dead Bee Heuristic to determine that the bug is caused by adding multiple line items. Remove a line item and the bug goes away. Add a second line item and the bug returns.
If there are 3 or 4 shipping or payment methods, try them each to rule them out. If there are 20, try a few.
Finally, use safety language in your bug description: "It **appears** that any time you submit an order with more than one line item, an “object reference not found” error is thrown."
There should be no question from your programmers as to whether shipping method matters because you have developed a cadence with them. They know that if shipping method mattered, you would have included it in the repro steps.
i always encouraged QA to summarize a one liner when logging a defect as a introduction. upon that, they then has the flexibility to shorten the steps and exclude the non requirement condition.
I have to agree with Eric here.
I usually submit bugs like this.
Dev knows if there were other relevant data, I would have included them.
Like, we usually add the login credentials. As not all the users has the same roles.
While I feel it's important to get the most relevant information in a bug report, I also try to remember that developers are not always the sole recipients of bug reports.
As a tester, I may have to come back to an older report looking for a duplicate or similar issue.
My business analysts may be monitoring the defect queue for future requirements refining.
Obviously I'm not going to throw in the kitchen sink (unless it's part of the cause) but making defect reports too terse can be a hindrance too.
Just my two cents.
While I feel it's important to get the most relevant information in a bug report, I also try to remember that developers are not always the sole recipients of bug reports.
As a tester, I may have to come back to an older report looking for a duplicate or similar issue.
My business analysts may be monitoring the defect queue for future requirements refining.
Obviously I'm not going to throw in the kitchen sink (unless it's part of the cause) but making defect reports too terse can be a hindrance too.
Just my two cents.
Eric,
I was having this exact discussion with a tester today.
If something is mentioned explicitly (payment method) then as a test manager and bug triage participant I automatically think that it's relevant (and crucial) to the reproduction of the issue. If all my testers (and devs) don't have this same expectation then I have some work to do. :)
I coach testers on looking for workarounds and noting them in a notes section. "User sees error" What state is the system left in? Does refresh fix it? Does it happen consistently or intermittently?
As well it's good to know if this used to work before and is broken now (is it a regression?)
Learning to distill a bug down to it's essence is one of many things that separate good testers from great testers. One has to learn to balance the time investigating non-essential data with the other priorities they have.
Something came to mind while writing this - is it non-required data or non-essential data. To you it might be non-required but to others it would be required. There might be an important difference in the wording there. It's hard to argue for putting non-essential data in a bug :)
Great writing and ideas - keep them coming!
Adam