…is the checks, stupid.

It doesn’t matter what framework you are using, what language you write them in, or how many you have.  What matters is how effectively your automated checks help determine ship decisions.

1. What should your automated checks observe?
2. What decision rules should your automated checks use to determine pass/fail? 

Those are the two hardest questions to answer.  You can’t Google them.  You can’t ask your testing mentors.  You’re tempted to hide your decisions when they’re poorly conceived (because you know few will ask).  You’re tempted to focus on what you know people will ask; the questions with the shortest answers:  How many automated checks?  Did they pass?

But, what are they checking?  You know that’s what matters.  Start building your automated check suite there.  The rest can follow.

In reference to my When Do We Need Detailed Test Cases? post, Roshni Prince asked:

“when we run multiple tests in our head… [without using detailed test cases] …how can we be really sure that we tested everything on the product by the end of the test cycle?”

Nice question, Roshni.  I have two answers.  The first takes your question literally.

• …We can’t.  We’ll never test everything by the end of the test cycle.  Heck, we’ll never test everything in an open-ended test cycle.  But who cares?  That’s not our goal.
• Now I’ll answer what I think you are really asking, which is “without detailed test cases, how can we be sure of our test coverage?”.  We can’t be sure, but IMO, we can get close enough using one or more of the following approaches:
1. Write “test ideas” (AKA test case fragments).  These should be less than the size of a Tweet.  These are faster than detailed test cases to write/read/execute and more flexible.
2. Use Code Coverage software to visually analyze test coverage.
3. Build a test matrix using Excel or another table.
4. Use a mind map to write test ideas.  Attach it to your specs for an artifact.
5. Use a Session Based Test Management tool like Rapid Reporter to record test notes as you test.
6. Use a natural method of documenting test coverage.  By “natural” we mean, something that will not add extra administrative work.  Regulatory compliance expert and tester, Griffin Jones, has used audio and/or video recordings of test sessions to pass rigorous audits.  He burns these to DVD and has rock solid coverage information without the need for detailed test cases.  Another approach is to use keystroke capture software.
7. Finally, my favorite when circumstances allow; just remember!  That’s right, just use your brain to remember what you tested.  Brains rock!  Brains are so underrated by our profession.  This approach may help you shine when people are more interested in getting test results quickly and you only need to answer questions about what you tested in the immediate future…like today!  IMO, the more you enjoy your work as a tester, the more you practice testing, the more you describe your tests to others, the better you’ll recall test coverage from your brain.  And brains record way more than any detailed test cases could ever hope to.

In my Don’t Give Test Cases To N00bs post I tried to make the argument against writing test cases as a means to coaching new testers.  At the risk of sounding like a test case hater, I would like to suggest three contexts that may benefit from detailed test cases.

These contexts do not include the case of a mandate (e.g., the stakeholder requires detailed test cases and you have no choice).

1. Automated Check Design: Whether a sapient tester is designing an automated check for an automation engineer or an automation engineer is designing the automated check herself, detailed test cases may be a good idea.  Writing detailed test cases will force tough decisions to be made prior to coding the check.  Decisions like: How will I know if this check passes?  How will I ensure this check’s dependent data exists?  What state can I expect the product-under-test to be in before the check’s first action?
2. Complex Business Process Flows:  If your product-under-test supports multiple ways of accomplishing each step in its business process flows, you may want to spec out each test to keep track of test coverage.  Example: Your product’s process to buy a new widget requires 3 steps.  Each of the 3 steps has 10 options.  Test1 may be: perform Step1 with Option4, perform Step2 with Option1, then perform Step3 with Option10.
3. Bug Report Repro Steps: Give those programmers the exact foot prints to follow else they’ll reply, “works on my box”.

Those are the three contexts I write detailed test cases for.  What about you?