Many (if not most) test teams claim to perform test case reviews. The value seems obvious, right? Make sure the tester does not miss anything important. I think this is the conventional wisdom. On my team, the review is performed by a Stakeholder, BA, or Dev.
Valuable? Sure. But how valuable compared to testing itself? Here are the problems I have with Test Case Reviews:
- In order to have a test case review in the first place, one must have test cases. Sometimes I don’t have test cases…
- In order for a non-tester to review my test cases, the test cases must contain extra detail meant to make the test meaningful to non-testers. IMO, detailed test cases are a huge waste of time, and often invaluable or misleading in the end.
- From my experiences, the tests often suggested by non-testers are poorly designed tests or tests already covered by existing tests. This becomes incredibly awkward. If I argue or refuse to add said tests, I look bad. Thus, I often just go through the motions and pretend I executed the poorly conceived tests. This is bad too. Developers are the exception, here. In most cases, they get it.
- Forcing me to formally review my test cases with others is demeaning. Aren’t I getting paid to know how to test something? When I execute or plan my tests, I question the oracles on my own. For the most part, I’m smart enough to know when I don’t understand how to test something. In those cases, I ask. Isn’t that what I’m being paid for?
- Stakeholders, BAs, or Devs hate reading test cases. Zzzzzzzzzz. And I hate asking them to take time out of their busy days to read mine.
- Test Case Reviews subtract from my available test time. If you’ve been reading my blog, you know my strong feelings on this. There are countless activities expected of testers that do not involve operating the product. This, I believe, is partly because testing effectiveness is so difficult to quantify. People would rather track something simple like, was the test case review completed? Yes or No.
16 comments:
Subscribe to:
Post Comments (Atom)
RSS
I agree with most of your points about test case reviews - especially with designing tests so that non-testers can understand what I do (devs are still an exception to this).
However I find the concept of a peer review (similar to that of a developer's code review) helpful in finding the flaws within my own test design, or even to see if I'm missing obvious points.
Also, I could use many of you testcase arguments for why I often don't like writing a formal test plan or test spec.
~ken
ksommerville,
You're right. Test Case Review is a kind of peer review. Maybe it's problematic when test cases are reviewed by those not considered "peers".
Code review? I know the correct answer is, "Sure, we do Code Reviews". I wonder, in practice, how well or how often it is done.
I see little benefit in anyone other than another tester reviewing testcases (except maybe devs)
I am a testing manager and I frequently review my staffs testcases mainly to check that they are well constructed as they are a document that lives on for a long time in our company and will be used by a number of other testers. I review them generally by actually running through them.
Are you arguing that in general test plans with documented test cases are not necessary? I agree that sitting through a test case review is no fun, and that it does take a lot of over head to properly document a test case. However, without that documentation how do you expect to perform regression tests, or have another QA pick up the project to pick up on the next iteration? Your post brings up some interesting questions.
I used to do this because, just like a lot of other things, it's what the established process said we should do. I grumbled about it, but I just did it. And I made the biz people sign off on it and I collected a little stack of papers with signatures that nobody ever looked at or cared about. Even the auditors seemed uninterested.
I don't do it anymore, of course. Rather, the biz people are responsible for the stories, and the acceptance tests associated with them. The testers on my teams are the ones that review *their* tests (and stories) to make sure they are complete and actually testable.
Of course, there are no signatures required.
Joe,
I'm not arguing that documented test cases are unnecessary. This is a separate discussion (but a good one to have). I'm just saying that maybe we should stop and think about how valuable these test case reviews are, instead of just doing them because that is what we do.
My own experiences tell me they are less valuable than good old raw test time.
We do test review, and developers attend the review meeting. Since we mainly do black-box testing, their feedback are very helpful. Sometimes we add more tests after the review.
Eric,
Great post. You mention a lot of common problems with test case reviews. I believe I have an interesting solution to most of your problems.
While I may be accused of bias as the founder of a testing tool company, please hear me out:
To your point that...
"* In order for a non-tester to review my test cases, the test cases must contain extra detail meant to make the test meaningful to non-testers. IMO, detailed test cases are a huge waste of time, and often invaluable or misleading in the end."
Please see
http://twitpic.com/i0r8y (Sample inputs to entered into Hexawise) and http://twitpic.com/i0rvd (Sample test cases generated by Hexawise)
To me, this is the right level of detail for many, but not all test case reviews. It gives reviewers enough detail for them to provide some real feedback but not so much to put them to sleep. You can get better feedback, faster. If you look closely, for example you could see that iPhone and Blackberry browsers are not tested. For that app, they definitely should be.
To your point that...
"* Forcing me to formally review my test cases with others is demeaning. Aren’t I getting paid to know how to test something? When I execute or plan my tests, I question the oracles on my own. For the most part, I’m smart enough to know when I don’t understand how to test something. In those cases, I ask. Isn’t that what I’m being paid for?"
... a second pair of eyes is often good. You don't want to spend too much time though. Provide less, more important info, not more (dull) info that will put them to sleep.
To your point that...
"* Stakeholders, BAs, or Devs hate reading test cases. Zzzzzzzzzz. And I hate asking them to take time out of their busy days to read mine."
... Ditto the point above.
To your point that...
"* Test Case Reviews subtract from my available test time. If you’ve been reading my blog, you know my strong feelings on this. There are countless activities expected of testers that do not involve operating the product. This, I believe, is partly because testing effectiveness is so difficult to quantify. People would rather track something simple like, was the test case review completed? Yes or No."
... largely agreed, but see also:
http://twitpic.com/i0s7l (Hexawise coverage chart) which is, you'll find with experience, often a very good measure of testing effectiveness (provided you and your reviewer were able to ensure that the right inputs got into the test case generator).
- Justin Hunter
Free version of Hexawise available here: http://www.hexawise.com/users/new
Eric,
For Test Case Reviews following are my points in its favour.
1) Peer Review of Test Cases would be useful if the peer tester needs to backup the tester down the road (contingency plan for attrition, tester leaving in future is known upfront).
2) Peer Review is useful if the peer will be executing the test cases in a subsequent cycle when more than one cycle is planned upfront. This serves better the implementation of "fresh eyes finds new faults".
3) It happens that test cases written by tester A is found lacking (during execution by tester B) in terms of test ideas, language/understanding, flow, logic and techniques employed and reviews are employed to prevent such occurrences.
4) Review by a Senior Tester / Test Lead is essential when the tester's knowledge on the domain is considered inadequate and/or the testers experience in all matters testing is well just starting.
5) I do not think reviews should be demeaning to one's intelligence / job function. Rather I think it is valuable education to know, learn and do better next time. Again Reviews can be in-depth for an inexperienced tester and a less in-depth or even a skim job for a more experienced tester.
6) The Business Analyst especially from the customer end would add lot of value if the same is part of scope. Not many are expected to know the business or the application supporting it better than him.
7) The test cases review by development does not happen more often than not in my experience. But the value may result in terms of requirements interpretation mismatch between the dev and test teams coming to light if the developer reviewer is diligent enough...
Alex,
I love your 2nd paragraph. I hadn't thought about that. It seems like it would remove most of the problems discussed in this post. The tester wins b/c they like poking holes in other people's stuff. Also, the tester understands the business domain language (or needs to), but the stakeholder does not need to understand the testing language.
Hmmm. I think this may take place at a primitive level on my team during feature review meetings.
Perhaps, with some tweaking, I can find a middle ground.
Sandeep,
Those are nice advantages. The stuff about doing reviews so that other testers can understand your tests is a hard sell for me (unless we have unlimited time).
- If I plan to test FeatureA, I will execute my own tests.
- If another tester must step in to test FeatureA, I would expect them to write/execute their own tests. If they use my tests as a crutch, they may not be testing effectively.
- If they are forced to execute my tests (at gun point), and they can't understand my tests because we did not conduct a formal test case review, they should either ask me about my test now or they should stop trying to be testers.
Fair enough?
I agree with you on the part that having test cases prior to have some one reviewing them is not always possible. I have experienced that even if someone(BA etc..) is asked to review my test cases I get a fair answer of if they are right or something like that. In most of the cases I have never gotten more answers on any missing tests in my list. So my conclusion is that there is no better person to review our test cases than another QA person. So if we ever have time we should plan to get them reviewed by another tester and i think that would help.
Good points about test case review, especially the fact that it is very time consuming.
At the same time you want feedback from many, and you want to give information to developers early.
The solution can be to start with one-liner test ideas. They are fast to write, and fast to review; and granularity can be adjusted to meet the audience.
I will cover this at EuroSTAR 2009: http://www.eurostarconferences.com/conferences/session-details.aspx?sessionId=131
My company used to have a formal test case review meeting with testers, developers and product management. It was a complete waste of everyone's time because generally no one other than the testers had any input. We have since gone to a one on one test review between testers and developers for some projects. The one benefit I have found from these is that as a tester, I am finding out changes in requirements before I start testing.
Eric,
I understand your frustration and you are right in some of those counts. Those meetings are DEAD Boring..and its harder than anything to get anyone other than QA in those meetings.
Im my position, I usually do the reivews, I would love to have a developer or a BA in that meeting. but the reality is I would have a hard time getting them in there. So I review it with the tester myself. That being said. I'ts not meant to be de-meaning. From my perspective its just meant to try and find Gaps.. I mean thats what we do right? we do quality. Everyone misses stuff, self included. This is just another set of eyes to try and help find some of that before it occurs. btw. Developers do reviews too.. its called a "code review". So don't feel bad man. It's nothing to do with you personally. Its just a good way to try and find gaps.
In my team we just do this activity internally with QA team, it's very helpful.
Agree with your point of view.