If you read my previous post, you know I help test a product on iteration 80. This product has few automated regression tests (we are working to change that). Currently, my test team faces regression testing armed with nothing more than their brains, their hands, and a big ticking clock.

Each 4 week iteration provides us with a little less than a week to spend regression testing. If you can relate to my situation, you may be interested in two painless practices we have adopted to better handle our ever increasing challenge of regression testing.

1. Group Regression Test Planning – We get the devs, BAs, and testers into a room and spend an hour brainstorming, to determine the top priority regression test areas of focus. We scan through the iteration's changes and draw on each other’s knowledge to create a draft of what will become the priority “1” tests or areas of focus. This occurs on a shared MS Excel regression test list spreadsheet. The prior iteration’s regression tests drop in priority...and so on.
   
   
2. Group Regression Test Sessions – We fill a large bowl with chocolate and invite the BAs and devs to join the testers in our classroom for two 3-hour sessions of group regression testing. All participants track their progress on the shared regression test list and we tackle it as a team. The team approach also occasionally shakes out multi-user-scenario-bugs, since our product is heavily dependent on user locking.
   

Our approach is not to complete a set amount of regression tests before we ship. Instead, we complete as many tests as we can, within the time provided. The prioritized test list makes this possible and shields us from things outside of our control.

Is anyone else doing regression testing with mostly manual testing? How do you pull it off?

Testing updates to old software is much more difficult than testing updates to new software. The older it gets, the more cobwebs form in lonely corners, which drop out of test rotation and concern.

One of my software projects is on its 5th year of steady updates (iteration 80). In software years, this product is 80-years-old. It makes me think of my house, which happens to turn 80-years-old this year.

Seemingly easy upgrades to my old house often result in way more changes than initially planned. When my wife talked me into replacing the cracked pink bathroom tiles, I soon uncovered 6” deep concrete poured between joists that had all but rotted through (due to 79 years of direct contact with concrete.


In the 1920's, I’m sure floating concrete floors were an excellent choice for tiled floors. And fortunately, the old growth pine joists were strong enough to hold the weight…but not for 80 years.

One thing leads to another and before we knew it, we were rebuilding our entire bathroom, including the floor.


Then the changes start to expand way out of scope. The walls will not be compatible with the new floor, so those will have to be replaced (but maybe we'll just patch and replace the worst parts of the outside wall).

Then you realize older technologies are not compatible with newer. You mean Home Depot doesn’t sell Quest plumbing fittings anymore? You mean I have to replace my polybutylene pipes with copper just to move my toilet?

Meanwhile, back at work, my users were asking for the seemingly easy upgrade to allow filtering in the product’s massive control-center-like grid. Sure, no problem…oh, hold on…we can do it but the current grid won’t support it so we’ll have to rebuild the entire module, a testing nightmare (depending on your perspective).

So be careful with work estimates on seemingly simple changes to older products. And try to remember all the often ignored, dependent systems that are affected.

Yesterday a manager told me they were troubled by a showstopper bug that was found after my team had finished testing a particular build.

(My team uses the “Time’s Up!” heuristic to determine when we’re done testing.)

Then the manager asked me what I could do to prevent this from happening again. I thought long and hard on this question and just when I was about to concede defeat, the ultimate solution came to me.

It may take several iterations, but if my team can build a time machine, we can guarantee no showstopper bugs will escape. I think it will really shorten our test cycles too. We’ll just throw stuff to the users, see what problems they encounter, then travel back in time and log it as a Showstopper...or maybe we'll call it a "Show Stopped". Better yet, we’ll just explain it to dev before they write it.

I just have to work out the predestination paradox, so I don’t get stuck in a causal loop, eternally traveling back in time to prevent the same bug from occurring.


My favorite recent time travel movies:

• Triangle
• Primer
• Timecrimes

Your application under test (AUT) probably interfaces with external systems. Fact: these external systems will go down at some point while a user is attempting to use your AUT.

Here is the obvious test:

1. Take ExternalSystemA down. If this is outside your control, simulate ExternalSystemA’s outage by changing where your AUT points for ExternalSystemA.
2. Trigger whatever user operations cause your AUT to interface with ExternalSystemA.

Expected Results: The user gets a friendly message indicating some functionality is blocked at this time. The support team gets an error alert indicating ExternalSystemA is not responding.



We executed the above test for 6 or 7 external systems and got our AUT robust enough to only block minimum functionality and provide good communication to users and the support team. However, just when we were getting cocky, we encountered a slight variation on the first test that crippled our AUT. Here is the test we missed for each external system.

1. Put ExternalSystemA into a state where it is up and running but cannot respond to your AUT within the amount of time your AUT is willing to wait. Note: We were able to simulate this by taking down ExternalSystemB, which gets called by ExternalSystemA.
2. Trigger whatever user operations cause your AUT to interface with ExternalSystemA.

Expected Results: The user gets a friendly message indicating some functionality is blocked at this time. The support team gets an error alert indicating ExternalSystemB is not responding.