Automating outside the box

Posted by Ben Simo

test

1. any standardized procedure for measuring sensitivity or memory or intelligence or aptitude or personality etc • the test was standardized on a large sample of students
2. the act of testing something
3. the act of undergoing testing • he survived the great test of battle
4. trying something to find out about it

automation

1. the act of implementing the control of equipment with advanced technology; usually involving electronic hardware • automation replaces human workers by machines
2. the condition of being automatically operated or controlled • automation increases productivity
   
3. equipment used to achieve automatic control or operation • this factory floor is a showcase for automation and robotic equipment

What is test automation?

I just read some marketing literature from some leading test automation tool vendors. According to one of the vendors, their tool supports the following: Novice testers can create robust and easily maintainable tests that mimic real-life use of an application with a few mouse clicks; and the automation tool will troubleshoot errors without human intervention. (I'd give the actual text if it didn't give away the vendor. You may be guessing correctly as you read this.) If this is test automation, I want some. This is what the tool vendors are telling the executives that authorize spending large sums of money. And when such claims are believed, a very high standard is set for test automation. Most testers that implement these tools quickly learn that the automation nirvana promised by tool traffickers is not available at any price.

Wikipedia currently has a decent definition for test automation -- if you read the whole thing. It starts out with a classic definition of test automation...

Test automation is the use of software to control the execution of tests, the comparison of actual outcomes to predicted outcomes, ... Commonly, test automation involves automating a manual process ...

There are many processes in testing that are good candidates for some form of automation if we do not try to remove the cognitive aspects of testing. Automation that retraces steps that have already been executed manually and reports "pass" or "fail" is unlikely to find bugs or help testers improve their understanding of a software system under test. The Wikipedia definition for test automation includes the following important aspect.

Another important aspect of test automation is the idea of partial test automation, or automating parts but not all of the software testing process. If, for example, an oracle cannot reasonably be created, or if fully automated tests would be too difficult to maintain, then a software tools engineer can instead create testing tools to help human testers perform their jobs more efficiently. Testing tools can help automate tasks such as product installation, test data creation, GUI interaction, problem detection (consider parsing or polling agents equipped with oracles), defect logging, etc., without necessarily automating tests in an end-to-end fashion.

I believe that partial test automation is not just an important aspect. It is essential. It is not possible to replace all aspects of a thinking human being with a machine. Test automation that helps automate testing tasks is likely to be a greater benefit than attempts at complete automation.

Instead of trying to create end-to-end test execution automation, think of how a doctor uses medical tests to help diagnose a patient's problems. No blood test or x-ray can diagnose or heal a patient. Doctors use the information reported by these tests in making a cognitive diagnosis. Look for ways that automated execution can help gather data that is useful in diagnosing software.

Test execution automation can be very useful -- but it may not be the best place to start.

Test automation can also be useful in generating test data and test cases. I believe the potential for automated test generation is often overlooked. Wikipedia mentions test generation automation yet implies that it is more academic than practical.

One way to generate test cases automatically is model-based testing where a model of the system is used for test case generation, but research continues into a variety of methodologies for doing so.

Pairwise testing has become a fairly common implementation of test generation automation. Combinations generated by a pairwise or other orthogonal array data generation tool can even be used for creating tests for manual execution.

I ask you to challenge your assumptions about test automation. Think beyond regression testing. Think beyond test execution. Look for ways that automation can help make you more efficient and put your automation efforts there first.

If you are a toolsmith, talk to and watch manual testers work. Look for ways that tools can help them do their work. You may find that the most beneficial automation has nothing to do with your initial assumptions about test automation.