Posted by Ben Simo
The point of philosophy is to start with something so simple as to seem not worth stating, and to end with something so paradoxical that no one will believe it."
- Sir Bertrand Russell
A heuristic is a commonsense guideline used to increase the probability of solving a problem by directing one's attention to things that are likely to matter. The word is derived from the Greek "heurisko" which simply means "I find". The exclamation "eureka", meaning "I found it!", shares roots with heuristic.
Just as the "Eureka!" screaming forty-niners of California's Gold Rush did not have secret knowledge or tools to tell them exactly where all the gold was buried; software testers don't know exactly where the bugs are going to hide. However, both software testers and gold miners know where bugs and gold have been found before. We can use that knowledge of past discoveries and the nature of what we seek to create heuristics that help us narrow in on areas most likely to contain the treasure.
Gold miners and testers can find treasure by accident. However, intentional exploration for bugs and gold are more likely to produce results than aimless wandering. That last statement is a heuristic. It is true most of the time, but sometimes it can be proven false. Sometimes wandering testers and miners stumble into something very important. I just don't want to do all my testing by accident.
Heuristic-based testing may not give us concrete answers, but it can guide us to the important things to test. Heuristics can also be used in automation to provide information to guide human testers.
There was a time that I told developers and project managers that I could not test their products when the requirements did not include straightforward "testable" criteria. I thought that I could not test without being able to report "pass" or "fail" for each test.
A good example was a requirement that stated something like "the user shall not have to wait an unacceptable amount of time". As a good quality school tester working in a factory school organization, I demanded to know how long was acceptable before I could start testing. I wanted to quantify "unacceptable". In this case, the truth is that "unacceptable" will vary based on the user. There were no contractual SLAs to satisfy. I may not be able to report that the requirement is met, but I can provide useful information to the project team to assist in determining if the performance is acceptable.
I have since learned that answers to heuristic questions are useful. It was in that same project that I started applying heuristics to automated data validation. Even without concrete requirements, we testers can provide provide information that is useful in answering important testing questions -- especially the qualitative questions. (As a side note, I am now amazed at how much we who call ourselves "Quality Assurance" like to focus on quantitative requirements and metrics.)
To use heuristics in testing, create a list of open-ended questions and guidelines. This will not be a pass/fail list of test criteria. It will not be a list of specific test steps. Instead it can be used to guide your test scripting and exploration for bugs. You will likely develop general heuristics that you can apply to all your testing and specific heuristics that apply to specific applications.
We need to be careful to apply heuristics as heuristics and not enforceable rules. For example, most people involved in testing web application have heard the heuristic that every page should be within three clicks of any other page. Applying this "rule" to web application design usually results in better usability. However, it does not always improve usability. Sometimes making every page within three clicks of another is not reasonable. Adding too many links are likely to confuse users more than they help. (Another heuristic?) Complex work flows often require that pages be more than three clicks away from others. Common sense needs to be applied to heuristics to ensure they are applied only when they fit the context.
Happy bug prospecting.
Eureka!
Next time.... apply heuristic oracles to test automation.
0 Comments:
Post a Comment