I attended Robert Sabourin’s “To Infinity And Beyond” class at STPCon. This guy’s sessions are fun because of his sense of humor and pop culture references. He played us a screen captured video of someone sending an email and we were asked to write down as many boundaries as we could think of within the video. I had nearly 40 in five minutes but many testers got more.
The point was to start a conversation about boundary tests that go beyond the obvious. The first take-away I embraced was a simple mnemonic Robert gave us for thinking about boundaries. I usually hate mnemonics because most of them are only useful as PowerPoint slide filler. However, I’ve actually started using this one:
“Take AIM”
A – Application logic
I – Inputs
M – Memory storage
These are three ways to think about boundaries for test ideas. Most features you attempt to test will have these types of boundaries. The other cool thing about this mnemonic is it prioritizes your tests (i.e., test the application logic first, if you have it).
The typical example is testing a text box. But let's apply it to something different. If we apply Robert’s mnemonic to the spec, “The system must allow the user to drop up to four selected XYZ Objects on a target screen area at the same time”, we may get the following boundary tests.
Application logic (i.e., business rules)
- Drop one XYZ object.
- Drop four XYZ objects.
- Attempt to drop five XYZ objects.
Inputs
- Attempt to drop one ABC object.
- Attempt to drop a selection set of three XYZ objects and one ABC object.
- Can the objects get to the target screen area without being dropped? If so, try it.
Memory storage
- Attempt to drop 1,000,000 XYZ objects (there may be a memory constraint in just evaluating the objects)
- Refresh the screen. Are the four dropped XYZ objects still in the target screen area?
- Reopen the AUT. Are the four dropped XYZ objects still in the target screen area?
What other boundary tests can you think of and which AIM category do they fit into?
RSS
I feel as though the memory storage one was put in there just to make the acronym instead of the other way around.
Secondly, anything will break under memory constraints.
Lastly I like pineapple chunks and am wondering why I'm wasting time writing this comment. Must unsubscribe from Eric's blog, but don't want to miss out on chance to harass him... sigh decisions, decisions.
Sure, anything will break under memory constraints. The trick is to handle it gracefully, so the user can continue their work.
If you search our bug DB for memory related errors, you'll see several, most of which were found in production using real world data. This has been a weak area of testing. We don't have enough time to test with realistic data...or we're too lazy.
Hmm.. I thought more about this and I suppose you're right, and memory issues are something we need to work on handling better.
However doing so is extremely difficult since memory exceptions are generally fatal and non-recoverable. Since if you get an OutOfMemoryException then anything you do in the handler such as putting up a message box, etc. will only attempt to use up more memory and thus exacerbate the problem.
I guess the difficulty in handling it was why I was a bit exasperated in the first post about it, and the real solution is to avoid having to handle it in the first place which is easier said than done.