If you want to have test automation
And don't care about trials and tribulation
Just believe all the hype
Get a tool of each type
But be warned, you'll have serious frustration!

(a limerick by Dorothy Graham)

I attended Dorothy Graham’s STARCanada tutorial, “Managing Successful Test Automation”.  Here are some highlights from my notes:

• “Test execution automation” was the tutorial concern. I like this clarification; sets it apart from “exploratory test automation” or “computer assisted exploratory testing”).
• Only 19% of people using automation tools (In Australia) are getting “good benefits”…yikes.
• Testing and Automating should be two different tasks, performed by different people.
• A common problem with testers who try to be automators:  Should I automate or just manually test?  Deadline pressures make people push automation into the future.
• Automators – People with programming skills responsible for automating tests.  The automated tests should be able to be executed by non-technical people.
• Testers – People responsible for writing tests, deciding which tests to automate, and executing automated tests.  “Some testers would rather break things than make things”.
• Dorothy mentioned “checking” but did not use the term herself during the tutorial.
• Automation should be like a butler for the testers.  It should take care of the tedious and monotonous, so the testers can do what they do best.
• A “pilot” is a great way to get started with automation.
• Calling something a “pilot” forces reflection.
• Set easily achievable automation goals and reflect after 3 months.  If goals were not met, try again with easier goals.
• Bad Test Automation Objects– And Why:
• Reduce the number of bugs found by users – Exploratory testing is much more effective at finding bugs.
• Run tests faster – Automation will probably run tests slower if you include the time it takes to write, maintain, and interpret the results.  The only testing activity automation might speed up is “test execution”.
• Improve our testing – The testing needs to be improved before automation even begins.  If not, you will have poor automation.  If you want to improve your testing, try just looking at your testing.
• Reduce the cost and time for test design – Automation will increase it.
• Run regression tests overnight and on weekends – If your automated tests suck, this goal will do you no good.  You will learn very little about your product overnight and on weekends.
• Automate all tests – Why not just automated the ones you want to automate?
• Find bugs quicker – It’s not the automation that finds the bugs, it’s the tests.  Tests do not have to be automated, they can also be run manually.
• The thing I really like about Dorothy’s examples above, is that she helps us separate the testing activity from the automation activity.  It helps us avoid common mistakes, such as forgetting to focus on the tests first.
• Good Test Automation Objectives:
• Free testers from repetitive test execution to spend more time on test design and exploratory testing – Yes!  Say no more!
• Provide better repeatability of regression tests – Machines are good checkers.  These checks may tell you if something unexpected has changed.
• Provide test coverage for tests not feasible for humans to execute – Without automation, we couldn’t get this information.
• Build an automation framework that is easy to maintain and easy to add new tests to.
• Run the most useful tests, using under-used computer resources, when possible – This is a better objective than running tests on weekends.
• Automate the most useful and valuable tests, as identified by the testers – much better than “automated all tests”.