Who tests the automated tests? Well, we know the test automation engineer does.  But per our usual rationale, shouldn’t someone other than the test automation engineer test the automated tests?  After all, they are coded…in some cases by people with less programming experience than the product programmers.

We’re experimenting with manual testers testing automated tests on one of my project teams.  The test automation engineer hands each completed automated test to a manual tester.  The manual tester then executes the automated test.  At this point the manual tester is testing two things at the same time.  They are:

1. Testing the automated test and
2. testing whatever the automated test tests

Or, to be more precise, we can use Michael Bolton speak and say the tester is:

1. Testing the automated check and
2. checking whatever the automated check checks

Whatever you call it, during this exercise, it’s important to distinguish the above two activities.  If the automated test's execution results in a “Fail”, it doesn’t mean the test of the automated test fails…but it may.  Are you with me still?  An automated test’s execution result of “Fail” may, in fact, mean the automated test is a damn good test.  It may have found a bug in the product under test.  But that is completely up to the manual tester who is testing the automated test.  One cannot trust the expected result of an automated test until one has finished testing the automated test.

Thus, the tester of the automated test will need to evaluate the test somehow and declare it to be a good test.  They may be able to do this several ways:

• Manipulate the product to see if the test both passes and fails under the right conditions.
• Execute the automated test by itself then as part of the test suite to determine if the setup/teardown routines adapt sufficiently.
• Read the automated test’s code (e.g., does the Assert check the intended observation correctly?).
• Manually test the same thing the automated test checks.
• Manipulate the product such that the test cannot evaluate its check. Does the test resolve as “Inconclusive”?

It would be nice to have the luxury of time/resources to test the automated checks thoroughly but in the end, I suspect we will have to draw the line somewhere and trust the test automation engineer’s test of their check.  In the meantime, we’ll see where this gets us.