Is There A Problem Here?

Posted by Ben Simo

msn video

To use this product, you need to install free software

---

This product requires Microsoft Internet Explorer 6 with Microsoft Media Players 10 and Macromedia Flash 6 or higher versions, or Mozilla Firefox 1.5 with Macromedia Flash 8, or Safari 2.0.4 with Macromedia Flash 8. To download these free software applications, click the links below and follow the on-screen instructions.

Step 1: download firefox 1.5download firefox 1.5

Step 2: Download Macromedia Flash PlayerMacromedia Flash player is free to download.
If still having problems, uninstall Flash and then re-install Flash.

Once the installations are complete, reload this page.

Aggravation Testing

Posted by Ben Simo

An example:



How Long Do I Have To Wait?



A few hours? I don't have hours. I am sitting in the car using borrowed WiFi from a campground. I had to seek out Internet access to use software that came on a CD. I finally find Internet access and now it says I may have to wait a several hours. Can I abort if it takes longer than I have? What happens if I lose my internet access while the firmware update is underway?

I'm already frustrated with this device. I'm already frustrated with the software. I was hoping that a firmware update might fix bugs and usability issues on the device itself. I have reached the tipping point. This thing is going back to the store.

Terrified by Improvisation

Posted by Ben Simo

[Improvisational comedy] involves people making very sophisticated decisions on the spur of the moment, without benefit of any kind of script or plot. That's what makes it so compelling -- and to be frank -- terrifying. ... What is terrifying about improv is the fact that it appears utterly random and chaotic. It seems as though you have to get up onstage and make everything up, right there on the spot. But the truth is that improv isn't random or chaotic at all. ... Improv is an art form governed by a set of rules... How good people's decisions are under the fast-moving, high-stress conditions of rapid cognition is a function of training, rules, and rehearsal.

- Malcolm Gladwell, Blink: The Power of Thinking Without Thinking

Now, reread the quote above and replace improv with exploratory testing. See a connection? Just as improvisational theater may appear to be random and chaotic (although entertaining) to the ignorant, exploratory testing can appear to be random and chaotic to those that have been taught to rely on scripts. Good improv and exploratory testing is neither. There are rules -- heuristics.

Heuristics are rules of thumb that help solve problems. In improvisational comedy, there are rules. These are not hard rules that guarantee comedy. These are rules that skilled improv actors can use to help keep things funny. Sometimes these rules don't work and actors have to adapt. And, because they aren't following a script, they can adapt when things don't work out. Some parts of improv are scripted. I am a fan of the television show Whose Line is it Anyway. Each comedy sketch in this show is given a structure (think charter) to direct the improvisation. This structure defines and restricts (think script) specific components of each sketch while leaving the bulk of the activity open to each actor to adapt to what happens as the sketch plays itself out. While we do not see it on screen, I suspect that a great deal of training, rules, and rehearsal went into the production of Whose Line. The shows did suffer from an occasional guest participant (usually a trained script actor) that was not as skilled at improv as the regulars. However, other guests (sometimes not actors) who understand the rules of improv have helped produce some of the funniest sketches.

Good exploratory testing works in the same way. Skilled exploratory testers set out with a charter -- a goal for each testing session. Skilled exploratory testers use heuristics to help them learn about the systems they test. Skilled exploratory testers practice.

Improvisation can have scripted aspects and rules that guide it. It is not chaotic and random. It is smart people using simple rules to make quick decisions and adapt to a changing environment under pressure.

Don't be fooled by the green lights

Posted by Ben Simo

If we're uncertain about the reliability and value of code, writing more code seems like a highly fallible and paradoxical way to resolve the uncertainty.
- Michael Bolton

There is a disturbing trend in software testing. This is a trend towards redefining test to be code and testing to be a coding activity.

In his book, Testing Object Oriented Systems, Robert Binder writes "Manual testing, of course, still plays a role. But testing is mainly about the development of an automated system to implement an application-specific test design." While this book contains a great deal of useful information about test design and test automation, I wholeheartedly disagree with this statement -- and I make a living developing test automation.

I find Bob Martin's statements about manual testing being immoral to be very disturbing. I know that he is referring to manual scripted testing, but that's not what I hear repeated. I can also think of some scripted testing that requires thinking manual testers and would be immorally expensive or dangerous to automate. (Context matters.)

I am concerned when I hear Ken Schwaber refer to "QA" as people doing incredible things in a "hopeless profession". (I've also heard him praise smart testers.) I am also concerned by QA people trying to defend their role by becoming process police.

All too often, I hear and read the words test and testing being tossed around with the assertion that all things called tests are the same, and are therefore interchangeable. If all things called testing are equal, then I would be the first to lobby for replacing skilled testers with developers that can create code called tests. TDD may be a great tool for helping developers build whatever they decide to build, but it is not the same a testing focused on providing stakeholders with information about value. These are very different things. (See What is Software Testing? for a sampling of testing diversity.)

Developers create. Testers critique. Good developers test what they create. The Agile and TDD emphasis on developers testing their own work is wonderful. I believe that a developer who is good at testing their own work is more valuable than one that is not so good at testing. However, the idea that code-centered TDD can replace value-centered testing by skilled testers is bad.

Testing is much more than exercising code. It is about finding and communicating useful information about value with limited time and resources. At the heart of good testing is a thinking person that questions the software and the people designing and building the software. The tools of testing are secondary.

It is difficult for a person to be both creator and critic. These require different skills and focus. In my experience, leaning too much towards create or critique hampers the other.

As a teenager, I created a software management program for floppy disk based computers. I was proud of my creation. My creation took advantage of new technology and had features that similar products did not have. I tested my creation. I refactored my code many times. The code was clean. The program was fast. I used my creation on a daily basis for over a year. I shared it with close friends. I thought I had created something really cool. I submitted my creation to a company that was soliciting programs for publication. My baby was rejected. It was not rejected due to being poorly designed, coded, or tested. It was rejected because it was deemed to not be of value to enough of the publisher's customers. I may have discovered this earlier if I had requested input from more than my closest friends.

Just like an American Idol contestant that can't sing, we can save ourselves time, money, and embarrassment if we solicit the input of good critics before the world is watching.

I want great developers that can create beautiful music on my development team. I also want a few Simon Cowells and James Bachs to let us know when we may be fooling ourselves.