Don't be fooled by the green lights

If we're uncertain about the reliability and value of code, writing more code seems like a highly fallible and paradoxical way to resolve the uncertainty.
- Michael Bolton

There is a disturbing trend in software testing. This is a trend towards redefining test to be code and testing to be a coding activity.

In his book, Testing Object Oriented Systems, Robert Binder writes "Manual testing, of course, still plays a role. But testing is mainly about the development of an automated system to implement an application-specific test design." While this book contains a great deal of useful information about test design and test automation, I wholeheartedly disagree with this statement -- and I make a living developing test automation.

I find Bob Martin's statements about manual testing being immoral to be very disturbing. I know that he is referring to manual scripted testing, but that's not what I hear repeated. I can also think of some scripted testing that requires thinking manual testers and would be immorally expensive or dangerous to automate. (Context matters.)

I am concerned when I hear Ken Schwaber refer to "QA" as people doing incredible things in a "hopeless profession". (I've also heard him praise smart testers.) I am also concerned by QA people trying to defend their role by becoming process police.

All too often, I hear and read the words test and testing being tossed around with the assertion that all things called tests are the same, and are therefore interchangeable. If all things called testing are equal, then I would be the first to lobby for replacing skilled testers with developers that can create code called tests. TDD may be a great tool for helping developers build whatever they decide to build, but it is not the same a testing focused on providing stakeholders with information about value. These are very different things. (See What is Software Testing? for a sampling of testing diversity.)

Developers create. Testers critique. Good developers test what they create. The Agile and TDD emphasis on developers testing their own work is wonderful. I believe that a developer who is good at testing their own work is more valuable than one that is not so good at testing. However, the idea that code-centered TDD can replace value-centered testing by skilled testers is bad.

Testing is much more than exercising code. It is about finding and communicating useful information about value with limited time and resources. At the heart of good testing is a thinking person that questions the software and the people designing and building the software. The tools of testing are secondary.

It is difficult for a person to be both creator and critic. These require different skills and focus. In my experience, leaning too much towards create or critique hampers the other.

As a teenager, I created a software management program for floppy disk based computers. I was proud of my creation. My creation took advantage of new technology and had features that similar products did not have. I tested my creation. I refactored my code many times. The code was clean. The program was fast. I used my creation on a daily basis for over a year. I shared it with close friends. I thought I had created something really cool. I submitted my creation to a company that was soliciting programs for publication. My baby was rejected. It was not rejected due to being poorly designed, coded, or tested. It was rejected because it was deemed to not be of value to enough of the publisher's customers. I may have discovered this earlier if I had requested input from more than my closest friends.

Just like an American Idol contestant that can't sing, we can save ourselves time, money, and embarrassment if we solicit the input of good critics before the world is watching.

I want great developers that can create beautiful music on my development team. I also want a few Simon Cowells and James Bachs to let us know when we may be fooling ourselves.

Green Means Go?

Traffic control devices are used on roads to help regulate the flow of traffic. When I taught defensive driving classes, I ensured that each class included a discussion about these devices. It is imperative that all drivers understand what each device means. My first Driver License test (in Germany) required that I properly identify 94 of 100 different signs to pass. There was a time that each local governing authority created its own traffic control devices. However, it was not long after the automobile became common that governments began working together to standardize these safety-critical devices. While there is no universal standard that is really followed (the USA being one of the countries that differs from most), standardization within each country (and some continents) has led to safer streets and highways.

These devices include signs, signals, pavement markings, barricades, and policemen. These devices are tools used to control the flow of traffic. However, these devices cannot really control traffic -- except for massive barricades and armed policemen. They provide information to drivers but they cannot force drivers to be safe or legal. As can be seen here, sometimes drivers ignore the signals.

Traffic lights are mostly standardized around the globe. Green means go. Yellow means caution. Red means stop. ... except for the extraterrestrial visitor in the movie Starman who learned by watching a bad example.

Red means stop;
Green means go;
and Yellow means go very very fast!
- Starman

The red, yellow, and green traffic light colors have become common in software development, testing, and production monitoring. Traffic light colors are regularly used to report the status of projects, systems, and individual tests.

I like simple status indicators -- in context. One of the first test execution tools I helped create used smiley faces to indicate passed tests and fulfilled requirements. I currently use color coding to indicate status in the test automation I develop. Colors help me quickly find test results that need attention. Simple color coding helps communicate test results at a high level.

I like to see green. I don't like to see red. However, I am not a member of the Church of the Green Bar*. I do not worship the green light. I do not trust the green light. I find green lights and bars useful but wrong. Green lights remove the story from the status.

Essentially, all models are wrong, but some are useful.
- George E. P. Box

A green traffic light may tell us that it should be safe to go but it does not guarantee that it is safe to go. A green light does not indicate that the intersection is clear. A green light does not indicate that it is safe to drive through the intersection. Drivers need to wait for the green light but they still need to check the intersection, identify potential risks, and decide if they believe it is safe to drive through the intersection. A green light means go if it has been determined that it is safe to go.

In the same way, a passed automated test does not mean that the software is good. Green simply means that the coded criteria was met.

Every time we test software -- whether with human eyes and mind, or with automation -- we only monitor the things that we choose to monitor. A human tester may notice things about the quality of a product that are not scripted. Automated test execution will only notice things that are coded in the script -- no matter how many time we run the test. If a factor that might indicate a problem is not part of the automation's green/red (pass/fail) criteria, it will not turn the light (or bar, or text) green.

I also find that there is often a misunderstanding of what automation does and does not do. The coder of the automation may know what it does when they code it. But will they really understand what a green light does and does not mean six months later? How about a year later? Five years later? Do other testers understand what the automation does? Does management understand what the automation does do and what it does not do?

Software systems can easily become complex. Computers allow us mortals to create complex systems that are beyond our ability to fully understand. We testers seek out software problems based on what we understand. We cannot completely test the software. We use a variety of tools and approaches to learn as much as possible. However, we are unlikely to completely understand a complex software system.
- Ben Simo^

We mere mortals and the automation we create are unable to monitor every thing that might matter. Therefore I believe it is dangerous to conclude that green means good.

The same goes for project management and system monitoring systems that use quantifiable metrics to set status.

Traffic lights based on the judgment of the people involved in the project are better indicators of status. If a light is green because a person set it to green, that person should be able to tell me the story behind the decision to make the light green.

Beware automated traffic lights.

[Update]

Automated traffic light indicators in testing tools are only badometers+. A badometer tells us when something is suspected to be bad cannot tell us if it is good. Better traffic light indicators are set by people that consider the risks associated with the information reported by our tools.

So, Does green mean go? Yes, but only after a human being has judged it safe to go.


* I don't know who first coined the term, but I first heard it from Brian Marick.
^ Since I regularly quote other people, I think I am entitled to quote myself. :)
+ A term I think I first heard used by Gary McGraw. Or what it Kim Possible?

Ugly Babies: Another Reason Why We Need Testers

... I got on the train ... And I noticed that the woman across from me in the aisle had her baby with her. Ugly baby. Ugly baby.

From the other end of the coach comes this guy and he was very drunk and he was staring at the baby. ... And the guy said, "I'm looking at that ugly baby. That's a horrible looking baby lady. Where'd you get that baby from?"

And the woman said, "I don't have to take that!" And she snatched the emergency cord and the train came to a screeching halt, and the conductor came running in.

Now this was his moment. At this moment he represented the Pennsylvania Railroad. And he said, "what's going on here?"

And the woman said, "This man just insulted me. I don't have to spend my money and ride this railroad and be insulted. I'd rather walk."

And the conductor said, "calm down! Calm down! Madame there's nothing, nothing that the Pennsylvania Railroad will not do to avoid having situations such as this. Perhaps it would be more to your convenience if we were to ... let you sit somewhere else in the coach. And as a small compensation from the railroad ... we are going to give you a free meal; and maybe we'll find a banana for your monkey."

- Flip Wilson


We are like parents when it comes to our creations: we can be blind to ugly.

I am a believer in developer testing. Good unit testing is essential to good software. Practices like Test Driven Development (TDD) help ensure that software works from the start. Anyone that's been in the software development business for long knows that the cost of defects exponentially grows as a project progresses. Preventing and finding defects early should be a goal in every software project. Early testing by developers does not remove the need for testing by someone other than the coder.

There are a number of reasons for "independent" testing. These reasons often include references to requirements testing, end-to-end testing, integration testing, system testing, and independent verification. These reasons focus on testing things as complete systems or from a user's viewpoint. I have another reason: Creators can be blind to ugly.

Creators have an emotional attachment to their creation. This attachment can get in the way of objective evaluation. It is difficult to be both creator and critic. It is as if creativity and critique are opposite sides of a scale. We can be both, but increasing one can harm the other. Creators need to balance the two without allowing self-criticism to hamper creativity.

I think its good for creators to have an emotional attachment to the product of their thoughts and labor. External critics can help a creator improve their creation without stifling creative thinking.

Creating and critiquing are separate skills. When one group can focus on one and another group can focus on the other: together they are stronger (if the critics are more tactful than the drunk in Flip's story).

At least that's my opinion. Now I open the floor to the critics. :)

Poka-Yoke

Poka-Yoke is not a dance. Its not an event at a rodeo. Its not what my kids do to each other in the back seat of the car. Poka-Yoke is Japanese for "mistake-proofing". Poka-Yoke was developed by Japanese industrial engineer Shigeo Shingo. He realized that people cannot be expected to work like machines and consistently do everything the same way every time they do it. People make mistakes and poorly designed processes can make it easier for people to err. Poka-Yoke's goal is to make it difficult for people to make mistakes through mistake prevention and detection.

Prevention

Applied poka-yoke gives users warnings about incorrect behavior and directs users towards the correct behavior. Computer PS/2 keyboards and mice share the same physical connector design but the connectors are usually color-coded to indicate which device goes into which port on a computer. Some computing hardware is shipped with warning stickers on top of connectors telling users to read a manual or install software before plugging in the device.

Poka-Yoke also means stopping users from doing the wrong thing. Diesel fuel pump nozzles will not fit in a vehicle that requires gasoline. The ignition key cannot be removed from most cars with automatic transmissions if the car is not in "park". Most cars with manual transmissions cannot be started unless the clutch pedal is pressed. These safety features prevent users from making mistakes.

Detection

Some errors cannot be prevented or are too expensive to prevent. The application of poka-yoke demands that errors be detected when and where they occur so that action can be taken before mistakes become bigger problems. Modern space heaters will automatically shut off if they are kicked over. A great example of automatic error detection and correction is the SawStop table saw that automatically disengages when the blade touches something that conducts electricity -- such as fingers. (See the video below.)


http://SawStop.com


Poka-Yoke Applied to Software

Poka-Yoke has existed in hardware products for decades. Poka-Yoke has improved quality and safety of many devices we use daily. While I do not like the behavior-shaping constraints of poka-yoke applied to intellectual tasks, directing and constraining user behavior is essential for good software. I do not advocate application of poka-yoke to the development process. I do advocate applying poka-yoke thinking to every stage of the software development life cycle to improve the quality of the software products we produce. Designers should think poka-yoke. Coders should think poka-yoke. Testers should think poka-yoke. Thinking about usability can lead to fewer bugs.

We are human and there will be bugs. To the extent that quality assurance fails at its primary purpose -- bug prevention -- it must achieve a secondary goal of bug detection.
- Boris Beizer
Software Testing Techniques

Prevention

Keep it simple. Make it easy for users to identify the expected correct way to use the software. Warn them if they try to do something wrong. Don't overwhelm users with unnecessary options.

When the risk of users not following a warning is great, prevent users from doing bad things. Things like list boxes and radio buttons can prevent users from entering invalid data. Data input constraints keep users and the software on the expected path. The security risks in web applications increase the necessity to prevent users from doing what they are not supposed to do.

Detection

It is especially important to detect errors that get past the warnings and constraints and stop processes before errors develop into bigger problems. The earlier an error is detected the easier it is to recover. Bad data detected when it enters a system does not have a chance to cascade into the rest of the system.

Poke-yoke thinking can improve usability and prevent bugs.

---

Some Poka-yoke resources on the web:

• Mistake-proofing Example Wiki
• Using Poka-Yoke Techniques for Early Defect Detection, Harry Robinson
• John Grout's Poka-Yoke Page
  

Where No Confabulation Goes Untested

confer

verb

• have a conference in order to talk something over

The Conference of the Association for Software Testing (CAST) is coming this July.

I missed last year's conference but have heard great things about it. Based on all the wonderful things I've heard from those that were there, I am looking forward to this year's conference.

The CAST is different than most conferences where people sit and listen to someone present to an audience without public questioning of what is presented. AST encourages testers to test the presentations. Time is allowed for discussion at every presentation. Challenging ideas is encouraged. I could go on. However, I don't think I can push this conference any better than David Gilbert. Therefore, please take a look at David's blog post: CAST in stone.

CAST early bird registration ends this week. Register at
http://associationforsoftwaretesting.org/conference/registration.html

I hope to see you there.

Ben

Falsifiability Testing

The best experiments deduce an effect from the hypothesis and then isolate it in the very context where it may be disproved.
- Michael Kaplan & Ellen Kaplan,
Chances Are: Adventures In Probability

In Focusing on Falsifiability, Stuart Thompson writes the following about a tester-friend's statement that testers can add value from the start of a project if they understand the project and its direction.

"With a clear understanding of what the software was actually trying to do, his team was able to provide useful feedback to the developers even within the first couple of release cycles."

When testers know the problem that software is trying to solve, they can first focus on the things that matter. They can test the assumptions. They can identify contexts in which the assumptions and the implementation can be disproved. Project managers and developers are usually focused on the solution. Testers can help identify the new problems created by proposed solutions and provide information to help the team determine if the new problems are not as bad as the ones they solve.

EACH SOLUTION IS THE SOURCE OF
THE NEXT PROBLEM
We never get rid of problems. Problems, solutions, and new problems weave an endless chain. The best we can hope for is that the problems we substitute are less troublesome than the ones we "solve".

- Donald C. Gause & Gerald M. Weinberg,
Are Your Lights On? How to Figure Out What the Problem REALLY Is

As Stuart points out, testers often have difficulty proving their worth when they prevent bad things from happening. It is usually in hindsight that we see where bypassed testing could have helped.

An example came to my mind as I read Stuart's post.

A change was made to an application. The testers knew about the problem that forced an update to the software. However, the testers we not told the details of the proposed solution. Shortly before the planned release, the testers discovered that the solution created new problems that were worse than the problem solved. In addition to creating new problems, the solution only worked in one of many likely contexts.

The developers told the testers that the solution was good because the "business" people approved it. The "business" people trusted that the developers knew how to implement the business need. It appears that no one specifically tested the assumptions. No one tried to put the solution in contexts that it may not work. Early information sharing with testers likely would have exposed the new problems created by the solution before time was spent coding, testing, and then re-engineering the solution.

It is ironic -- although common -- that a decision made in haste due to time pressure delayed the update to the software. Time spent on QA and testing early is usually going to save time and money.

Had someone focused on the falsifiability of the solution, the problems caused by the solution would have been prevented before a single line of code was written.

I have heard testers complain that no one invites them to be involved early in the process. I too have joined in that chorus. A colleague recently reminded a group of testers that sometimes we aren't included early because we don't ask. Yes, sometimes getting involved early is as easy as asking.

Ask to be involved early. Seek out ways to focus on falsifiability instead of nit-picking incomplete implementations, and developers are likely to invite you back.

Not Gonna Bow

Individuals and interactions
over processes and tools

Working software
over comprehensive documentation

Customer collaboration
over contract negotiation

Responding to change
over following a plan

- Agile Manifesto

Neary 400 years ago, Francis Bacon challenged the status quo in scientific thought in “The New Organon”. James Bach recently pointed out some interesting quotes from this work that apply to software testers. I agree.

Bacon argued that placing our preconceived beliefs over what we observe causes great harm. He went so far as to describe these harmful preconceived notions as “idols”. Bacon put these idols into four categories:

• Idols of the Tribe: Errors common to mankind.


• Idols of the Cave: Errors specific to each individual’s education and experience.


• Idols of the Market Place: Errors formed through association with others — often due to misunderstanding others.


• Idols of the Theater: Errors formed from dogma (institutionalized doctrine) and flawed demonstrations.

All of these exist in software testing. As testers, we should be questioning these “idols”, not worshiping them. Sometimes questioning them may prove them right.

Bacon did not ask anyone to abandon their beliefs without cause. Instead he asks that we not make them idols capable of leading us to ignore what would be obvious if we weren’t looking through the distorted mirror of our idols.

A modern day simplification of Bacon’s arguments may be the Agile Manifesto. We should not let our idols of process, documentation, contracts, and plans prevent us from accomplishing the desired goal. Process, documentation, contracts, and plans are only good in as much as they help. They should not prevent us from seeking improvement.

In some ways I believe that the promotion of testing folklore is the result of an industry-wide desire to show that we are mature — as mature as the engineering of physical products. I believe that eagerness to demonstrate maturity helps lead to the implementation of bad processes and cerfifications. Ironically, enforced process (see the bottom of the FSOP cycle) works best for the immature and gives the impression that anyone that can follow the process can test software.

Don't get me wrong. Process and documentation are good things that help even the smartest people when appropriately applied.

"The only thing that interferes with my learning is my
education." - Albert Einstein

We need to seek continual improvement. It is sad that process and certification often become idols that overshadow the real goals.

Hey Dad, when I grow up, I also want to be square.

Erkan Yilmaz, a fellow tester blogger from Germany, recently pointed out that the slogans in my post "Slogans are models" may not transmit their message across languages and cultures. He attempted to guess at what some of the slogans meant without the context of American culture and advertising. These slogans that most Americans will instantly understand didn't work very well out of their context.

We both saw this as an example of how recipients of information do not always have the full context in which the information originated. As testers, we need to admit when we don't understand and seek the answers (and context) from those that know. Sometimes we may need to bring subject matter experts into the conversation to fully understand what we are testing. If you don't know, ask questions. If you think you know, ask questions. You are bound to learn something.

I recently sat in some presentations by people from the "business" (as in not IT) side of some projects in which I am involved. I learned a great deal about how customers use our products and the business' vision for the future of the products. This information will help me better test the products. It is good to know more about the context in which the products I test are used.

Know thy user, for he is not thee.
- David S. Platt

After our exchange about the American slogans, Erkan provided the following list of slogans from German-speaking countries for interpretation by those of us that live outside that context.

1. “Hey Dad, when I grow up, I also want to be square.”
2. “We wake up earlier.”
3. “We can do everything but speak Standard German.”
4. “With the second eye you see better.”
5. “Firm as a rock in the surge.”
6. “We demand and bring forward personalities.”
7. “Try it in a gentle way.”
8. “It was never so valuable as in these times.”
9. “If it makes you beautiful…”
10. “Well, is today already christmas ?”

What do you think these mean?

See Erkan's original post of this list here.
And after you have tried to interpret the list, look here for their real meanings.

Viel Spaß!

Ben

How Doctors Think

Michael Bolton recently brought Dr. Jerome Groopman’s newest book, “How Doctors Think”, to my attention. Michael suggested to the software-testing list that this book contains information that is relevant to software testing. In this book, Dr Groopman explores how doctors diagnose medical problems and what can be done to reduce bad judgment.

Michael Krasny recently interviewed Dr. Groopman on public radio about this book. You can listen to the interview here. Dr Groopman made a number of statements during this interview that I believe apply to us software testers’ search for and diagnosis of software bugs. I expect to find more in the book.

Here are some gems from the radio interview:

About computer-assisted diagnosis software.

It’s a two-edged sword. On one hand, it can be useful if you feed in the most important and relevant symptoms, but the problem is that you need to be able to illicit them effectively from the patient and so you return to language and you return to some of these thinking errors or these cognitive traps that we can all fall into. So if you put in for, example the first prominent symptom that you hear from someone into the software package, it may be that the fourth thing that the person complains about is the most relevant one. Recently you may have seen, for example, in reading mammograms, that computer assisted diagnosis actually generated more false positives. It caused the radiologist to think that shadows which he or she would normally think were benign were in fact cancer and didn’t add to the diagnosis of cancer –the true positives. So I think this technology is worth exploring, but I think that we have to be very careful about it because it some ways it can seem seductive. You cannot substitute a computer for a human being who listens and filters and judges and elicits from the patient the full story.

The same applies to test automation. It can be a useful tool if we don’t let it lead us astray. We human testers need to be actively involved in listening, filtering, judging, and eliciting the full story.

About time-constraints:

Its one of the biggest issues in the current delivery of medical care. You know, everyone is being told as physicians to see patients in 12 minutes. See patients in 11 minutes. So you’re sitting there with one eye on the clock and you can’t think well in haste. And there’s the natural inclination to anchor onto that very first impression as they stop there and just function as if you’re working on an assembly line in a factory. ... patients are being ill-served.
…
I think everyone in the medical system feels under siege. There’s not a lot of satisfaction but in a way this is penny-wise and pound-foolish because the cost of misdiagnosis is extraordinary. … It also costs in terms of dollars. Its much more
expensive to care for and treat an advanced problem than to make a diagnosis early on.

I think we have to force ourselves to resist and part of that can be done: sometimes we have to extend a visit. But we really are beholden to administrators and managed care plans, so one of the things that I’ve begun to do is if I can’t get to the bottom of a problem in my 15 minutes allotted visit,
then I say to the patient, you know, I haven’t figured it out. I need to think more. ... reschedule an appointment and spend more time.

Testers also make mistakes under time pressure. We also need to be aware that our first impressions may not be right. Testing is not a factory assembly line (even though this is a widely-held view). Testers need to be engaged throughout testing and not just mindlessly follow test scripts. And sometimes we need to lobby to spend more time than the schedule originally gives us.


About the art vs. science of diagnosis and improving the effectiveness of that diagnosis:

There’s a seductive nature of numbers, but statistics from clinical studies and so on are just averages and they may not reflect the individual in front of you. ... So real judgment involves customizing medicine: looking at scientific data but also seeing how it applies or doesn’t apply to the person sitting in front of you.
...
It’s not technology, but its language. Language is really the bedrock of medicine. Most of what we do with a doctor – or should be doing with a doctor – is talking… engaging in a dialog.
...
... all of us are wired to make these kinds of snap judgments to sort of go with our first impressions. to rely very heavily on intuition. … That often works but in medicine unfortunately too often it doesn’t work because we have a tendency to latch on to that first bit of information that we get from a patient and then run with it; as opposed to keeping our mind open and constantly doubting ourselves.

Testing is not an assembly line process. Testers need to keep their minds engaged throughout the testing process. We should not ignore our snap-judgements (blink testing), but we also need to look and think beyond those first impressions. We need to continually question both the software and our own judgement as we test.

We need to communicate more than numbers. We need to communicate stories. We need to translate bugs and metrics into language that matters to the business.

Get excited about the negative

"It is the peculiar and perpetual error of the human understanding to be more moved and excited by affirmatives than by negatives."

--Francis Bacon

Selective thinking is the practice of giving more weight and credence to information that confirms our beliefs than information that may contradict our beliefs. We are all guilty of this. We tend to easily believe data that confirms our beliefs and experience. We tend to ignore data that does not confirm our beliefs.

This confirmation bias can greatly impact our work as software testers. Testers (and developers) often test (and code) to confirm requirements are met by testing the positives. We often overlook the negatives.

Peter Wason's card problem demonstrates this. The problem involved four cards that each have a number on one side and a letter on the other. They are presented with the following values visible:

A

B

4

7

The other side of the cards is not shown.

The following claim is made: "If a card has a vowel on one side, then it will have an even number on the other."

The following question is then asked: Which cards do you need to turn over to determine if the above claim is true?

Try to solve the problem before continuing.

Research has shown that most people get the answer wrong. The majority of people believe that A and 4 must be turned over to answer the question. This suggests that most people try to confirm the positive when the question requires that we also try to disprove the statement. Both the positive and negative need to be confirmed to answer the question. Click here to see the answer.

When we test software we need verify both the positive and the negative of the requirements. We need to ensure that the software does what it should do and does not do what it should not do.

You're too negative

I believe that some level of pessimism is required to be a good software tester. Some of the best testers I have met are pessimistic towards the systems they test. These black hat testers (of which I am one) consider what might go wrong and ask questions. Although I believe that applied pessimism is a necessary for good testing, this negativity can hurt relationships and the project if it is not constructive. Optimist developers and project managers often have trouble understand us pessimists. The result of our pessimism needs to be better preparedness, not shared depression by all involved in a project. This is the type of pessimism that is the subject of Dr. Julie Norem's book, The Positive Power of Negative Thinking. Dr. Norem defines "defensive pessimism" as

Defensive pessimism is a strategy used by anxious people to help them manage their anxiety so they can work productively. Defensive pessimists lower their expectations to help prepare themselves for the worst. Then, they mentally play through all the bad things that might happen. Though it sounds as if it might be depressing, defensive pessimism actually helps anxious people focus away from their emotions so that they can plan and act effectively.

I do not agree that defensive pessimists are necessarily "anxious people". I see the pessimism as a necessary part of good critical thinking.

Think you might be a defensive pessimist? Take the defensive pessimist quiz.

While the pessimist black hat is a necessary part of testing, Julian Harty argues that we need to try on Ed DeBono's other hats as well: both in analysis and applied methodology. Take a look at Julian's CAST presentation from last year.

Expecting the unexpected. Part 2

How can we create automation that can deal with the unexpected?
The first step is to create test automation that "knows" what to expect. Most GUI test automation is built by telling the computer what to do instead of what to expect. Model-Based Automated Testing goes beyond giving the computer specific test steps to execute.

Model-Based Testing is testing based on behavioral models instead of specific test steps. Manual testers design and execute tests based on their mental models of a system's expected behavior.

Automated tests can also be defined using models instead of scripting specific test steps. Tests can then be randomly generated from those models -- by the computer instead of a manual tester. The computer can even recover from many errors that would stop traditional test automation because it knows how the system is expected to behave. And by knowing how it is expected to behave, it can detect unexpected behavior. Unexpected does not necessarily mean wrong behavior. The behavior could be wrong or it could be something that was not included in the model. The computer can report the unexpected behavior to human testers for investigation and future updates to the model.

For example, if one path to functionality to be tested fails, the MBT execution engine can attempt to access that functionality by another path defined in the model.

Of course, there will always be some cascading failures that stop both automated and manual tests. MBT inherently provides better error handling than scripted test automation.