Failure Story #2– This App Will Never Be Automated
Posted by Eric Jacobson at Tuesday, November 26, 2013Here’s another failure story, per the post where I complained about people not telling enough test failure stories.
Years ago, after learning about Keyword-Driven Automation, I wrote an automation framework called OKRA (Object Keyword-Driven Repository for Automation). @Wiggly came up with the name. Each automated check was written as a separate Excel worksheet, using dynamic dropdowns to select from available Action and Object keywords in Excel. The driver was written in VBScript via QTP. It worked, for a little while, however:
- One Automator (me) could not keep up with 16 programmers. The checks quickly became too old to matter. FAIL!
- An Automator with little formal programming training, writing half-ass code with VBScript, could not get help from a team of C# focused programmers. FAIL!
- The product under test was a .Net Winforms app full of important drag-n-drop functionality, sitting on top of constantly changing, time-sensitive, data. Testability was never considered. FAIL!
- OKRA was completely UI-based automation. FAIL!
Later, a product programmer took an interest in developing his own automation framework. It would allow manual testers to write automated checks by building visual workflows. This was a Microsoft technology called MS Workflow or something like that. The programmer worked in his spare time over the course of about a year. It eventually faded into oblivion and was never introduced to testers. FAIL!
Finally, I hired a real automator, with solid programming skills, and attempted to give it another try. This time we picked Microsoft’s recently launched CodedUI framework and wrote the tests in C# so the product programmers could collaborate. I stood in front of my SVP and project team and declared,
“This automation will shave 2 days off our regression test effort each iteration!”
However:
- The automator was often responsible for writing automated checks for a product they barely understood. FAIL!
- Despite the fact that CodedUI was marketed by Microsoft as being the best automation framework for .Net Winform apps, it failed to quickly identify most UI objects, especially for 3rd party controls.
- Although, at first, I pushed for significant amounts of automation below the presentation layer, the automator focused more energy on UI automation. I eventually gave in too. The tests were slow at best and human testers could not afford to wait. FAIL! Note: this was not the automators failure, it was my poor direction.
At this point, I’ve given up all efforts to automate this beast of an application.
Can you relate?
RSS
"The tests were slow at best and human testers could not afford to wait."
I don't get it- why were these too slow?
Also... My first impression here was the mistake of undervaluing and not motivating enough the programmer, that was involved in product and was volunteering in test automation framework development. I miss a sentence here describing why was it omitted after all :)
Damn you are so stupid..
1. If tests are slow.. use virtual machine to run them;
2. Keyword Driven Automation sucks. Why you spen so mutch time on this without researching another options;
3. Why you choose to use programming language whose isn't used in your company?
4. Don't forget.. automated tests need maintainment. It doesn't resolvs itself when developers does changes.
I'm using test automation for 2 years and it worked for me.
I never was able to understand real benefits of "Key Word Driven" framework. To me it was just a different format (EXCEL vs. flat file).
I have not heard about successful implementation of such framework on a small or mid-size project.
All "successful" implementations happened at large organizations with "Standards Driven" test approach where testers were required to automate thousands and thousands of test cases.
It looks like "Key Word Driven" framework would allow you to "effectively" automate that many test cases.
Disclaimer: I am not talking about test effectiveness. I am talking only about effectiveness of creation of thousands of automated test cases.
hahaha! I can relate for sure! I´m using Coded UI for a BIG application, knowing nothing about coding at the beginning. Now I have a rookie dev with me. We had all the issues you mention, but somehow we could overcome those. For example, the 3rd party controls...we have Infragistics like, everywhere, and it was a real challenge, but now we can handle those bad guys perfectly. The performance was also enhanced recently by changing some Search Criteria and Filters on the properties of the methods. However...we are 2 persons for a hell of an application, that has major releases often...and add to that the fact that I´m being asked to execute and add more value by selling our automation scripts to other teams!
Anonymous,
It's comments like yours:
"Damn you are so stupid"
...that explain why more testers don't tell test failure stories. What a shame.
A lot of what you said is true.
I was initially told to use uiautomation framework - huge investment of my time - and later we swopped this out for ui automation framework - fail.
The build cycle become more frequent and the 200 or so usable tests I pushed out - ever single last one of them broke at some point. When trying to fix the broken ones between cycles, a new build would come out before the last ones got fixed - often leading to nothing more than a constant cycle of broken tests. Virtually for 1 year hardly any value.
Now we're this close to getting the tests running in new engine and MS Agent is so damn unreliable during playback that we need to solve this now too.
All in all everyone is losing patience. It's been fun and a great learning curve but actual value to company hasn't been very high.