'Twas the night before prod release, when all through the build,
Not a unit test was failing, the developers were thrilled;

The release notes were emailed to users with care,
In hopes that new features, soon would be there;

The BA was nestled all snug in her chair,
With visions of magnitude ready to share;

And I on my QA box, trying not to be stressed,
Had just settled down for a last minute’s test;

When during my test there arose such a clatter,
I opened the error log to see what was the matter;

And what to my wondering eyes should appear,
But an unhandled fault, with its wording unclear;

When I showed it to dev, and he gave me a shrug,
I knew in a moment it must be a bug;

More rapid than eagles, dev’s cursing it came,
And he shouted at testers, and called us by name;

“Now, Jacobson! Now, Zacek! Now, Whiteside and Surapaneni!
On, Cagle! On, Addepalli, on Chang and Damidi!

Stop finding bugs in my web service call!
Now dash away! dash away! dash away all!"

And then, in a twinkling, I heard from the hall,
The tester who showed me, scripts can’t test it all;

As I rejected the build, and was turning around,
Into my cube, James Bach came with a bound;

He was dressed really plain, in a baseball-like cap,
And he patted my back for exploring my app;

He had a big white board and a little round belly,
That shook when he diagrammed like a bowlful of jelly;

He was chubby and plump, a right jolly old elf,
And he laughed when he saw RST on my shelf;

Then he spoke about testing, going straight to his work,
And attempted traspection, though he seemed like a jerk;

His eyes -- how they twinkled! his dice games, how merry!
He questioned and quizzed me and that part was scary!

He told me of lessons he taught at STARWEST,
Made an SBT charter and then told me to test;

Then I heard him exclaim, ere he walked out of sight
“Happy testing to all! ...just remember I’m right!”

Yesterday we had to roll back some production code to its previous build. It was essentially a performance problem. The new bits kept timing out for the types of requests users were making from the system.

Our poor, battered support team had to send out a user notice that said something like “Sorry users, for all the timeouts. We have just rolled back the code until we can fix the problems. Oh and by the way, that means you’ll no longer have these new features we just gave you.”

Shortly thereafter, the head of my department, indirectly asked me why these issues were not caught in testing. …This is the stuff nightmares are made of. I felt like throwing up, resigning, or crying. Once my heart started beating again, I realized I had logged a bug for these problems. Exhale now.

The bug was fairly accurate, stating the users would get timeout errors if they attempted anything beyond a specific threshold. During a bug meeting, it was down-graded in priority and the team decided to move forward with the release. In hindsight, there was a bit of group-think going on. We were so anxious to get the release out, we figured the users would put up with the performance problems. Boy were we wrong.

Being a tester is scary. At any given moment, all hell may break loose in production. And when it does, asking why the tester didn’t find it is, of course, the fairest of questions.

I brainstormed with my report dev and we came up with two valuable JART expansions.

The first is coded and has already found two bugs (and saved me from days of bordom). Comprehensive data checking in the report output is complicated but what about checking for data that is missing? My reports display the string “[Not Found]” when data fields get mapped incorrectly or other datasource problems occur. It only took about an hour to write a function that scans all report results looking for the string. Since JART already exported each report result to a text file, I simply automated Notepad’s Find functionalty to look for specific strings passed into my SearchReportResultsForString function. This was way easy. Now JART checks about 2240 distinct data columns across 113 reports for this issue.

I use said function to also look for the “No Data Found” string. If I find this string it means the report results returned but other than the cover page, no data matching the search criteria exists. This check gets reported as a “Warning” or inconclusive result. I use it to help me determine when I need to adjust my filter criteria parameters on each report.

The second expansion is to build a routine that saves off the report results of each iteration as “Iteration N Baselines”. Then, using the same report filter criteria and same data store, save off the next iteration’s report results as “Iteration N+1 Baselines”. Once JART has at least two iterations of baselines, JART will compare the files from each baseline to ensure nothing has changed. If I can pull this off, it will be HUGE. I’m expecting it to only support about 75% of my reports. The other 25% is time sensitive data that may not have a way to remain frozen in time.

Yesterday, a dev pointed out that the reports display the report execution date (i.e., the current date) in the header and footer. Thus, they should never match. Ooops! I think I can still work around it, but it does complicate things a bit.

My former manager and esteemed colleague asked me to teach a two hour class about Session Based Testing (SBT). We had tried SBT a couple years ago, when I was fresh out of Michael Bolton’s excellent Rapid Software Testing course.

I was nervous as hell about the class because most of the testers I work with were signed up and I knew this was an opportunity to either inspire some great testing or look like a fool. So I spent several weeks researching and practicing what I would teach. I decided an Exploratory Testing (ET) primer was necessary for this audience before SBT could be explained properly.

ET proved to be the most intimidating subject to explain. Most of what I found was explained by members of the Context-Driven School (e.g., James and Jon Bach). Nearly everything I found NOT explained by members of the Context-Driven School was heavily criticized (by members of the Context-Driven School) for not being true ET. With all this confusion over what ET actually is, one wonders how well the Context-Driven School has actually explained what they mean. I found various statements from videos, blogs, papers, and my RST courseware that ranged from...

• It’s a technique…no it’s a method…no it’s a “way of testing”.
• It’s the opposite of scripting…no, it can used with scripting too, even while automating.
• All testers use ET to some extent…no wait, most testers aren’t using it because they don’t understand it.
  

After (hopefully) explaining ET, I was easily able to transition into SBT, making the case that SBT solves so many of the problems introduced by poorly conducted ET (e.g., lack of artifacts and organization). I explained the essential ingredients of SBT:

• Time Boxing
• Missions
• Capturing Notes, Bugs, and Issues
• Debriefing
  

Then I demonstrated my favorite SBT tools:

• Session Tester
• The Scan Tool
  

In the end, about half the audience appeared luke warm while the other half appeared skeptical to confused. I blame it on my own delivery. I think more light bulbs went off during the ET section. SBT takes a bit more investment in thought to get it.

For myself, however, the class was a success. Ever since my research, I’ve actually been using SBT and I love it! I also have some better ideas on how to teach it if I ever get a second chance. Special thanks to Michael Bolton and James Bach, who continue to influence my testing thoughts in more ways than anyone (other than myself).