Why does test automation often fail to deliver?

Posted by Ben Simo

Software Is Automation

Testers are continually asked to test more in less time. Test automation is often considered to be a solution for testing more in less time. Software is automation. It makes sense to automate the testing of software. However, functional black box test automation rarely does more in less time. Why does automation seldom deliver faster, better, and cheaper testing?


Common Test Automation Pitfalls

The following problems are often encountered during test automation projects.

1. Tests are difficult to maintain and manage.

Many GUI test automation attempts to replace just a single step of the manual testing process: test execution. This can work well when there is value in repeatedly executing the same steps and the application is stable enough to run the same script multiple times. However, in the real word, applications change and doing the same thing over and over again is rarely beneficial. A thinking human tester still needs to create the tests, code the automation, review the results, and update the automation code every time the system changes. The development, maintenance, and management of automated test scripts often requires more time and money than manual test execution.

2. Test results are difficult to understand.

Test results need to be manually reviewed. Automated test results often do not contain enough information to determine what failed and why. Reviewing results takes time. Manually repeating the tests to determine what really happened takes even longer. Insufficient results reporting can easily negate any advantages of automation.

3. Application changes and bugs can prevent tests from completing.

Application changes and bugs that will not stop a human tester can easily stop automated test execution in its tracks due to cascading failures. A single failed step in an automated test can easily prevent execution of all later steps. Updating and restarting tests every time they encounter the unexpected is not an improvement over manual testing.

4. Tests retrace the same steps over and over, and don’t find new bugs.

Scripted automation will repeat the same steps every time it is run. It will not encounter bugs that are not on the pre-cleared scripted path. Sometimes consistency is good, but consistency will not find new bugs. Many testers make the mistake of believing that their automated tests are doing the same things as a good manual tester. Scripted automation will only do what it is coded to do.

Functional black box test automation frequently requires more manual work to poorly do less than a human tester. This is the result of attempting to duplicate manual testing with automation. Attempting to duplicate manual testing processes not only fails to reduce costs or improve coverage; it creates additional manual work.

Useful software rarely mirrors the manual process that it replaces. Word processors are much more than virtual typewriters. Spreadsheet programs are more than virtual calculators. The process needs to change to take advantage of the strengths of the machine. People and machines have different strengths. A machine cannot think. A human tester is unlikely to be happy running tests all night or performing tedious calculations. Most software is built to assist human users, not replace them. Test automation should assist human testers, not attempt to replace them.


Rules of Test Automation

Arguing that computers cannot think like a human being, James Bach proposed the following test automation rules in a blog entry titled Manual Tests Cannot Be Automated.

Rule #1
A good manual test cannot be automated.

Rule #1B
If you can truly automate a manual test, it couldn’t have been a good manual
test.

Rule #1C
If you have a great automated test, it’s not the same as the manual test that you believe you were automating.

A great automated test is one that assists testers by doing what is not easily done manually without creating more manual work than it replaces. Great automation goes beyond test execution by assisting with test generation and providing useful information to manual testers. Combining Model-Based Testing (MBT) with a tester-friendly automation framework is one way to improve the effectiveness of test automation.