Talkin' 'bout test in a differrent light

Pullin' out my big black book
Cause when I need a word defined that's where I look
So I move to the L's quick, fast, in a hurry
Threw on my specs, thought my vision was blurry
I look again but to my dismay
It was black and white with no room for grey
Ya see, a big "V" stood beyond my word
And yo that's when it hit me, that luv is a verb.

Words come easy but they don't mean much
When the words they're sayin' we can put trust in
We're talkin' 'bout love in a different light
And if we all learn to love it would be just right.

- DC Talk


The DC Talk song "Luv is a Verb" points out that love is something to be acted out. Real love is action, not just words and feelings. Love is expressed through action.

Like love, the word test is both a noun and a verb. Also like love, test requires action. Even the noun definitions for test describe action.

test

noun

• trying something out to find out about it
• any standardized procedure for measuring sensitivity or memory or intelligence or aptitude or personality, etc.
• a set of questions or exercises evaluating skill or knowledge
• the act of undergoing testing
• the act of testing something

verb

• put to the test, as for its quality, or give experimental use to
• test or examine for the presence of disease or infection
• examine someone's knowledge of something
• show a certain characteristic when tested
• achieve a certain score or rating on a test
• determine the presence or properties of (a substance)
• undergo a test
  

from Princeton WordNet

While I don't think we'll find anyone that argues that test is not a verb, people involved in software development seem to use it primarily as a noun. I have nothing against the many things we create that we call tests. Our test cases, test code, test charters, and whatever test things we create can be useful tools -- but they are not the test.


Let's think about test in a different light.

Several years ago, Shrini Kulkarni challenged me in questioning whether there can be such a thing as an automated test. I don't think I disagreed with Shrini. I've not been one to trust testing to machines, but I've been a fan of automation throughout my testing career. I've automated many testing tasks, but not believed I can automate the testing itself.

Earlier this year, Michael Bolton told me of a distinction he was thinking about between checking and testing. While I had no disagreement with this distinction, I wasn't thrilled with the terms. I wanted something more descriptive. I thought Michael was making a distinction I had been trying to make: a distinction between validation and investigation. I've since come to understand that Michael is making a slightly different distinction. Michael has recently written a series of blog posts better describing the Checking vs. Testing distinction. Michael has limited the scope of checking to observations and decisions rules that can be executed without sapience -- without a brain-engaged human. If something requires human sapience, it is testing, not checking.

Yesterday, an insightful tester, Lanette Cream, made a nice attempt at defining test on her blog. In her latest revision, she defines test as follows.

A test is
an action
which produces discoveries
that can be used to evaluate product quality.

I like that this definition identifies action, discovery, and evaluation as being core to testing. However, I'm thinking of pushing, or rather constraining, this just a bit further.

What if we were to say that the evaluation is the action and discovery is the goal?

A test would then be the sapient part of validation or investigation -- the thinking and learning that cannot be automated. All those other things we do to test are really support activities that help us evaluate.

Test is not a document. Test is not code. Test is not executing a program. Test is not applying a procedural decision rule. Test is not anything that can be done by a machine. Test is the act of evaluating. Test requires sapience.

Test is thinking and learning that leads to discovery. We may test by evaluating existing data. We may test by running experiments that produce new data. We may take the output of automated checks to test. We may provide what we learn as input to coding new automated checks. The test is the action we perform in our minds.

This may come across as nitpicking vocabulary. That's not my intent. My goal is not to limit anyone's definition of test, but rather to shed a different light on what I believe sets testing apart from checking, and gives both checking and testing value.

If a check fails in a forest and no one is around to hear it, does it make a sound?

The true value of our checking and testing is in the mind of a sapient tester. What value is there in all the things we call checks and tests without a tester (whatever their role or title) evaluating information and learning?

Test is sapient evaluation that leads to discovery

.

I'm not quite comfortable with this. I want the emphasis to be on the sapient activity; and not generating and collecting data to support the thinking without ignoring that it is a necessary part of testing.

Regardless of where we shine the light or draw lines, let's keep in mind that test is a verb.

What do you think? Testing of my half-baked ideas is welcome and appreciated.

Regular Expressions

(bb|[^b]{2}); [Tt]hat is the \?\.

Regular expressions are great tools for testers. I have found them useful for describing GUI objects to GUI test automation tools. I have found them useful for automation results validation. I have found them useful for extracting data I care about from voluminous log files. I've also found them useful for manipulating data.

What are regular expressions? Regular expressions are patterns for finding text of interest. They are supported by many test tools, system utilities, text editors, and programming languages.

Regular expressions can include the following meta characters to define patterns.

• ^ Matches the beginning
  
• $ Matches the end
  
• . Matches any single character
  
• * Matches zero or more occurrences of the preceding character
  
• \ Escape character
  
• ? Matches zero or one occurrence of the preceding character
  
• + Matches one or more occurrences of the previous character
  
• [ ] Defines a character class
  
• [^ ] Defines an exclusion-based character class
  
• \{ \} Matches a specific number or range of instances of the previous character
  
• \( \) Treats the expression between \( and \) as a group
  
• | Or. Use to match one of many expressions
  
• \< Matches the beginning of a word
  
• \> Matches the end of a word
  
• \b Word boundary
  
• \B Not a word boundary

* Many tools do not support all meta characters

Here are some example regular expressions:

“frog”

• Matches “frog”, “bullfrog”, and “tree frog”; but not “Frog”

“^Frog”

• Matches “Froggy went a courting”, but not “Quality Frog”

“frog$”

• Matches “frog”, “bullfrog”, and “tree frog”; but not “froggy” or “The frog sat on a log.”

“.at”

• Matches “cat”, “rat”, “bat”, “goat”, and “gnat”

“20*5”

• Matches “2005”, “20005”, “20000000000000000000000005”, “25”; but not “2ABC5” or “2006”

“Spee?d”

• Matches “Sped” and “Speed”; but not “Speeed”

“20+5”

• Matches “2005” and “20005”, but not “25”

“200[5-9]”

• Matches “2005”, “2006”, “2007”, and “2009”; but not “2004”

“199[0-9]|200[0-9]”

• Matches years 1990 through 2009.

“[0-9][0-9]*\.[0-9][0-9][^0-9]”

• Matches “1.29”, “1.29%”, and “1234.55”; but not “1.299” or “.29”

“A[LKRSZ|C[AOT]|D[CE]|F[LM]|G[AU]|HI|I[ADLN]|K[SY]|LA|M[AFRHINOPST]|N[CDEHJMVY]|O[HKR]|P[ARW]|RI|S[CD]|T[NX]|UT|V[AIT]|W[AIVY]“

• Matches any valid 2-letter US postal state or territory name abbreviation.

Want to learn more?

Take a look at my slides from last night's presentation to the Denver Mercury User Group. Check out Wikipedia. Or try a Google Search. If you ask bb|[^b]{2}, check out Think Geek.

Ha[p]{2}y T[ea]sting\.

Arranging Abstract Absolute Artifacts

For any system of interesting size it is impossible to test all the different logic paths and all the different input data combinations. Of the infinite number of choices, each one of which is worth of some level of testing, testers can only choose a very small subset because of resource constraints.
- Lee Copeland,
A Practitioner's Guide to Software Test Design

The complexity of software makes it impossible to test all the possible things we could test for all but the most simple systems. (And I have often argued that even very simple systems cannot be completely tested.) This inability to test everything requires that we testers (and testing developers) identify the things that we believe are most likely to help us fulfill our testing mission. This makes test design very important. We not only need to design our tests in a way that supports our mission -- we need to communicate our testing in a way that supports our mission.

There are many ways that we can design and document tests: from very high level exploratory testing charters to the very specific step-by-step procedures of scripted automation. We often need to communicate a single test using various levels of detail.

A high-level test charter might be fine for communicating to project managers but we may need to describe step-by-step tasks when we document how to reproduce a bug found during exploratory testing. (Tools like Test Explorer can help document exploratory testing.)

High-level test execution steps may be fine for some manual test execution but these steps need to be made explicit for automation. And sometimes the details matter for tests executed by humans.

Detailed test procedures aren't very good for communicating functional coverage to product owners or managers. Sometimes we need to think about even the most scripted tests at a high level and not get bogged down in the details.

Sometimes we need to communicate tests designed and defined at a high level with great detail. Other times we need to communicate low-level automated tests at a high level. Different levels of detail are required by different people at different times.

There was a great deal of discussion at the Agile Alliance Functional Testing Tools Visioning Workshop about the desire to easily define tests using a variety of levels of abstraction and to communicate tests in different ways for different people. We considered how tools could be built to support the disparate needs of people involved in software development.

Elizabeth Hendrickson nicely summed up how tools can help support this by providing "A Place To Put Things". I am all for separating the essence of tests from automation code. Tools like xUnit and FIT aren't great because they do good testing. In fact, these tools don't really do the testing. They are useful tools because they give people a place to put things. When we have a place to put things, we are better organized. When we are better organized, we can communicate better.

Having a place to put things helps keep our testing organized and helps us communicate -- whether we are documenting tests as examples, designing FIT tests, scripting GUI automation, or documenting exploratory testing ideas.

One group of us at the workshop broke off to discuss the things that we testers need a place to put. We considered the possibilities of defining parts of tests at different levels of abstraction.

What if we could easily define tests at the highest possible (or reasonable) level of abstraction and then add details only when and where details are required?

What if a test could be defined at a high enough level that automated test execution engines could run the same tests on different platforms, or with different user roles, or with different data?

We did a little brainstorming and wrote down things we use to document a test and then divided these into three categories -- or levels of abstraction: business requirements (goals), interaction design (activities), and implementation design (tasks). Some items ended up in the twilight zone -- between or occupying multiple levels.

Business Requirements (Goals)

• Goal
• Expectation
  
• As a ... I want to ... so that ...
• Exploratory testing charter

Interaction Design (Activities)

• Present / Communicate Results
• Communicate test to users, dev, business
• Actor
• Domain objects
• Action
• User Preferences
• wait for so long
• set up pre condition
• model-based test generation
• Given; When; Then
• Wait until
• orchestration
• Roles
• Time Passes...
• Branding
  

Twilight Zone (Somewhere crossing over activities and tasks?)

• Domain Models
• Verify
• System state
• data
• state transitions
• user state

Implementation Design (Tasks)

• objects
• Check Results
• Show
• STATES
• Do ...
• Control GUI, API, test harness
  

I'm sure that there are many other things we testers would like a place to put that we didn't think of in our few minutes of brainstorming.

Traditionally, defining tests at various levels of abstraction has been difficult. I've seen people try to add abstraction to tests and spend more time maintaining and documenting the various abstractions levels than I think the benefits were worth. I've also successfully used abstraction in test automation to make the same tests executable on multiple platforms.

If we can find the right abstractions to communicate the intent of an example, we might be able to finally break free of the perception of functional tests as brittle, hard-to-understand, write-only artifacts. Even better, we might find a way to layer new tools on top of these abstractions so that, if I want to write my examples in plain text and you want to drag boxes around on a screen and she wants to use the UML, we can each use the form that speaks most clearly to us.
...
I want more names for my common things. I want to deal in goals and activities not checkboxes and buttons. I want to give the system a few simple bits of information and have it tell me something I didn't know. I want to show my examples to everyone in the project community and have them lift up their understanding rather than drown it in permutations and edge cases and "what happens if the user types in Kanjii?".
- Kevin Lawrence,
In Praise of Abstraction

Another group at the workshop worked on devising a framework to give us a common place to put things. If we had a common place to put things, then a variety of tools could use the same data and users could select whatever tools best work for their needs -- and desired level of abstraction. Thanks to Elizabeth for clarifying what I think many were thinking but did not express so clearly: we need a place to put things.

Whether you are trying to create a one-size-fits-many testing framework or a specialized tool to support a specific need: first develop places to put things.

Given the infinite testing possibilities, the best testing tools are those that help us organize, understand, and communicate our tests.

Green Means Go?

Traffic control devices are used on roads to help regulate the flow of traffic. When I taught defensive driving classes, I ensured that each class included a discussion about these devices. It is imperative that all drivers understand what each device means. My first Driver License test (in Germany) required that I properly identify 94 of 100 different signs to pass. There was a time that each local governing authority created its own traffic control devices. However, it was not long after the automobile became common that governments began working together to standardize these safety-critical devices. While there is no universal standard that is really followed (the USA being one of the countries that differs from most), standardization within each country (and some continents) has led to safer streets and highways.

These devices include signs, signals, pavement markings, barricades, and policemen. These devices are tools used to control the flow of traffic. However, these devices cannot really control traffic -- except for massive barricades and armed policemen. They provide information to drivers but they cannot force drivers to be safe or legal. As can be seen here, sometimes drivers ignore the signals.

Traffic lights are mostly standardized around the globe. Green means go. Yellow means caution. Red means stop. ... except for the extraterrestrial visitor in the movie Starman who learned by watching a bad example.

Red means stop;
Green means go;
and Yellow means go very very fast!
- Starman

The red, yellow, and green traffic light colors have become common in software development, testing, and production monitoring. Traffic light colors are regularly used to report the status of projects, systems, and individual tests.

I like simple status indicators -- in context. One of the first test execution tools I helped create used smiley faces to indicate passed tests and fulfilled requirements. I currently use color coding to indicate status in the test automation I develop. Colors help me quickly find test results that need attention. Simple color coding helps communicate test results at a high level.

I like to see green. I don't like to see red. However, I am not a member of the Church of the Green Bar*. I do not worship the green light. I do not trust the green light. I find green lights and bars useful but wrong. Green lights remove the story from the status.

Essentially, all models are wrong, but some are useful.
- George E. P. Box

A green traffic light may tell us that it should be safe to go but it does not guarantee that it is safe to go. A green light does not indicate that the intersection is clear. A green light does not indicate that it is safe to drive through the intersection. Drivers need to wait for the green light but they still need to check the intersection, identify potential risks, and decide if they believe it is safe to drive through the intersection. A green light means go if it has been determined that it is safe to go.

In the same way, a passed automated test does not mean that the software is good. Green simply means that the coded criteria was met.

Every time we test software -- whether with human eyes and mind, or with automation -- we only monitor the things that we choose to monitor. A human tester may notice things about the quality of a product that are not scripted. Automated test execution will only notice things that are coded in the script -- no matter how many time we run the test. If a factor that might indicate a problem is not part of the automation's green/red (pass/fail) criteria, it will not turn the light (or bar, or text) green.

I also find that there is often a misunderstanding of what automation does and does not do. The coder of the automation may know what it does when they code it. But will they really understand what a green light does and does not mean six months later? How about a year later? Five years later? Do other testers understand what the automation does? Does management understand what the automation does do and what it does not do?

Software systems can easily become complex. Computers allow us mortals to create complex systems that are beyond our ability to fully understand. We testers seek out software problems based on what we understand. We cannot completely test the software. We use a variety of tools and approaches to learn as much as possible. However, we are unlikely to completely understand a complex software system.
- Ben Simo^

We mere mortals and the automation we create are unable to monitor every thing that might matter. Therefore I believe it is dangerous to conclude that green means good.

The same goes for project management and system monitoring systems that use quantifiable metrics to set status.

Traffic lights based on the judgment of the people involved in the project are better indicators of status. If a light is green because a person set it to green, that person should be able to tell me the story behind the decision to make the light green.

Beware automated traffic lights.

[Update]

Automated traffic light indicators in testing tools are only badometers+. A badometer tells us when something is suspected to be bad cannot tell us if it is good. Better traffic light indicators are set by people that consider the risks associated with the information reported by our tools.

So, Does green mean go? Yes, but only after a human being has judged it safe to go.


* I don't know who first coined the term, but I first heard it from Brian Marick.
^ Since I regularly quote other people, I think I am entitled to quote myself. :)
+ A term I think I first heard used by Gary McGraw. Or what it Kim Possible?

Problems: So What's On All Those Sticky Notes?

In my previous post about the Agile Alliance Functional Testing Tools Workshop , I wrote the following:

After reviewing existing tools used by agile teams: we identified software testing issues that have been solved (yellow), those that have been partially solved (orange), and those that have not been solved (pink). As I recollect, most of the solved issues were technical problems and most of the unsolved problems were people problems. Many of the partially solved problems were those for which I believe we have technical solutions but have not yet been integrated and presented in ways that best support people.

In case you are wondering what problems we wrote down on these notes, Frank Maurer kindly transcribed them for the workshop participants and I have posted them below.

Looking at this list reminds me of the traffic safety problem lists I made in the defensive driving classes I used to teach. As unsafe driving practices were brought up by students, I would add them to a list on the whiteboard. I then asked the students to identify whether each item on my list was primarily due to driver skill or driver attitude. The students usually blamed most of the problems on driver attitude.

Skill and attitude play important roles in software development and testing. Team members need both. Brian Marick addresses this in his guiding values of discipline, skill, ease, and joy. Instead of looking at the functional testing problems as skill or attitude problems, I looked at them as man or machine problems. I asked myself if each problem appeared to be mainly a human problem or a technical problem.

Software development and testing involves a mix of people and technical problems. Interfacing people with people and people with technology is often harder than interfacing technology with technology. Most of the identified problems have both technical and human aspects to the problem and possible solutions. Some are due to the nature of people or the nature of software and will likely never be completely solved.

I find that identifying whether a problem is primarily a people or technology problem helps me identify possible solutions. I quickly scanned this list and identified whether I thought things were primarily human or technical issues. My notes are included to the right of each item. I don't necessarily interpret each item as its author (a human communication problem), and I do not necessarily agree that each item is a problem or belongs in the specified group.

As you review this list, ask yourself if the problem and possible solutions are grounded in people or technology.

Unsolved	Define Test	Human
Unsolved	Organizing large sets of Tests/Expect. Actions/Examples for a large, complex system so you can wrap your head around the whole thing.	Human
Unsolved	Having tests survive handoffs. Project team -> op support -> proj team.	Human and technical
Unsolved	Getting people to care	Human
Unsolved	Transferability of ubiquitous language to other projects	Human
Unsolved	Write SW that is understood	Human
Unsolved	Reducing Uncertainty	Human
Unsolved	Limitations of natural language	Human
Unsolved	How would we act if we really believed code was an asset?	Mostly Human
Unsolved	Allowing Customers to articulate their expectations in a format/tool/way that is comfortable for them	Mostly Human
Unsolved	Multi-model specification text + table + graphic in one test.	Mostly Technical
Unsolved	Domain experts	Human
Unsolved	Fully Automated regr. That does not reduce dev.velocity.	Mostly Technical
Unsolved	Generate a domain model from tests.	Human and technical
Unsolved	Testing usability as part of acceptance testing in incremental development.	Human
Unsolved	Common language to express GUI based tests.	Mostly Human
Unsolved	Conveying "experts" perspective to majority of development team.	Human
Unsolved	Automated Software Development	Technical (Machines aren't creative)
Unsolved	Functional tests that can be easily re-used later in lifecycle.	Mostly Technical
Unsolved	Test business requirements independent of current interaction/Api design.	Mostly Human
Unsolved	Composing tests into useful larger tests	Technical and Human
Unsolved	Test first performance	Human
Unsolved	Getting BA to write the tests	Mostly Human
Unsolved	Having customer to be able to write test and enjoy it.	Mostly Human
Unsolved	Different test notations for different user groups.	Human problem, technical solution?
Unsolved	Acceptance/Functional Tests good for communication and automation.	Human problem, technical solution?
Unsolved	Model the time domain " and 3 months later an email.	Mostly Technical? (Don't want execution to take 3 months.)
Unsolved	Change touch-point dynamically.	?
Unsolved	Terminology (Test or not a Test?)	Human
Partially Solved	Understand what has not been tested.	Human and Technical
Partially Solved	Trace tests into project management tools	Human and Technical
Partially Solved	Getting buy-in for need to automate.(docs,tests,specs)	Human and Technical
Partially Solved	Accurately & completely communicating requirements.	Human problem, partial technical solutions
Partially Solved	Satisfying every role's need/desire to be at center, in control.	Human
Partially Solved	Having functional tests specify requirement specifications.	Human and Technical
Partially Solved	Sustain a productive conversation with all stake holders.	Human problem, partial technical solution
Partially Solved	How do we get across what the project would feel like if things were going well.	Human
Partially Solved	Write robust (U.I) tests that are not brittle.	Mostly Technical
Partially Solved	Fragile tests.	Mostly Technical
Partially Solved	Valuing individuals and interactions over processes and tools.	Human
Partially Solved	Describing customer intent.	Human
Partially Solved	Executable (as tests) Models (as specifications)	Human and technical
Partially Solved	Test partitioning	Human and technical
Partially Solved	Tests as support artifacts.	Human and technical
Partially Solved	Test Generation Automation.	Human and technical
Partially Solved	Running tests parallely ( Fast feedback)	Mostly technical
Partially Solved	Reconciling preferred style of abstraction.	Human and technical
Partially Solved	Dealing with size.	Human
Partially Solved	Finding the right words in which to write a test.	Human
Partially Solved	Express requirement in the domain language, graphical, word based , table based.	Mostly Technical
Partially Solved	Functional Test driven development..not just for agilists. How to sell to waterfallists?	Mostly Human
Partially Solved	How to Integrate tools?	Mostly Technical
Partially Solved	Common test case format.	Human and Technical
Partially Solved	Cooperation and collaboration between tool developers (tool stack, tool platform).	Human and Technical
Partially Solved	Change from one notation to another. (graphic -> tabular)	Mostly technical
Partially Solved	Test/Example -> model (generated model based tests)	Mostly technical
Partially Solved	IDE for testers and BAs	Human and technical
Partially Solved	Get all roles actively involved	Mostly Human
Partially Solved	View specs/examples/tests, differently for different roles.	Mostly Technical
Partially Solved	Different editors for different roles?	Mostly Technical
Partially Solved	Super-power IDE	Technical solution to human problems?
Partially Solved	Test Refactoring	Human and technical
Partially Solved	Refactoring tests	Human and technical
Partially Solved	Prioritize and execute tests to get faster feedback.	Mostly Human
Partially Solved	Describe a test at an appropriate level of abstraction	Mostly Technical
Partially Solved	Choosing what to automate (when you can't automate everything)	Mostly Technical
Partially Solved	Tools to support exploratory testing	Human and Technical
Partially Solved	Reusable test artifacts (poor modularity cohesion)	Mostly Technical
Partially Solved	Build community with BAs	Mostly Human
Partially Solved	Allow for refactoring from /to code <-> tests <-> req'ts	Mostly Technical
Partially Solved	Setup a wiki to discuss smaller problem solving.	People
Partially Solved	Capture war stories + testimonials + experiences.	Human, partial technical solution
Partially Solved	Test Maintenance	Human and Technical
Partially Solved	Book: Patterns of Self testing software	?
Partially Solved	Shared vocabulary around parts of a functional testing solution ("Fixture",etc)	Human
Partially Solved	Ensure adequate test coverage.	Mostly Technical
Partially Solved	Communication of what has been tested	Human and technical
Partially Solved	Communication using the ubiquitous language.	Human
Solved	Express automated/able tests in tables	Mostly Technical
Solved	Correctly implementing programmer intent	Mostly Technical
Solved	Deliver SW to test that doesn’t crash immediately	Mostly Technical
Solved	Provide traceability between Story or Requirement and accpetance/functional test	Mostly Technical
Solved	Gui Testing - functional testing is more than GUI testing	Technical
Solved	Data-Driven Testing	Technical
Solved	Driving Apps	Technical
Solved	Integrating test executors/drivers with build process	Technical
Solved	Edit Fit tests from eclipse	Technical
Solved	Report Results	Technical to report, human to be understood
Solved	Express expectations in code	Human and technical
Solved	Unit testing	Mostly Technical

I think we can solve the technical issues and use technical solutions to help people manage some of the people problems. Applying discipline and skill to solve the technical problems may help add to testers' ease and joy.

Where do you think the solutions lie? Have a solution? Please share it.

Better Tools for Individuals through Collaboration

Individuals and
interactions

over
processes and tools

- from the Agile Manifesto

I spent the second half of last week at the Agile Alliance Functional Testing Tools Visioning Workshop. (How's that for a long name?) Before the workshop, I was thinking that it seemed a little oxymoronic to have an agile workshop with a focus on tools. Perhaps my thinking was triggered by my concerns about those who seem to value "agile" processes and tools (often ones they sell) more than people.

Agile people are supposed to care about people and not care about tools. Right? Wrong.

while there is value in the items on
the right, we value the items on the left more

- from the Agile Manifesto

Software is developed by people for people. Agility involves building better software by adapting to the needs of people instead of letting processes and tools lead the way. Process and tools do best when they have a supportive role in software development. Better tools can support agility but they cannot make anyone agile.

The tool-centric discussions at the workshop were driven by a desire to build better software for people that build and test software. It is about people.

It then seems quite apropos that the book I indiscriminately grabbed off the shelf (well, I picked it for its size more than its content) to read on the airplane to and from the workshop is Ben Shneiderman's Leonardo's Laptop: Human Needs and the New Computing Technologies. The first chapter contains the following paragraphs that affirm my thinking about the role of automation in software testing. (Emphasis is mine.)

The first transformation from the old to the new computing is the shift in what users value. Users of the old computing proudly talked about their gigabytes and megahertz, but users of the new computing brag about how many e-mails they sent, how many bids they made in online auctions, and how many discussion groups they posted to. The old computing was about mastering technology; the new computing is about supporting human relationships. The old computing was about formulating query commands for databases; the new computing is about participating in knowledge communities. ...

The second transformation to the new computing is the shift from machine-centered automation to user-centered services and tools. Instead of the machine doing the job, the goal is to enable you to do a better job. Automated medical diagnosis programs that do what doctors do have faded as a strong research topic; however, rapid access to extensive medical lab tests plus patient records for physicians are expected, and online medical support groups for patients are thriving. ... Natural language dialogs with computerized therapists have nearly vanished, but search engines that enable users to specify their information needs are flourishing. The next generation of computers will bring even more powerful tools to enable you to be more creative and then disseminate your work online. This Copernican shift is bringing concerns about users from the periphery to the center. The emerging focus is on what users want to do in their lives.

- Ben Shneiderman, Leonardo's Laptop

Although many think of us software testers and developers as eccentric nerds, software developers and testers are human too. Like other humans, we desire tools that help us do a better job. This was the theme of the workshop: envisioning ways that tools can help us do a better job testing software.

After reviewing existing tools used by agile teams: we identified software testing issues that have been solved (yellow), those that have been partially solved (orange), and those that have not been solved (pink). As I recollect, most of the solved issues were technical problems and most of the unsolved problems were people problems. Many of the partially solved problems were those for which I believe we have technical solutions but have not yet been integrated and presented in ways that best support people. Much of the "what's next" discussion at the workshop was focused on how to integrate existing tools that each partially solve problems but together could move problems to the solved group.

Once the technical problems are solved, we can work on the tools to help with the people problems: we can move from old computing to new computing.

In Leonardo's Laptop, Ben Shneiderman presents a framework for integrating creative activities of people. This framework for mega-creativity consists of four activities:

• Collect: Learn from what exists
  
• Relate: Consult with peers and mentors
• Create: Think: explore solutions
  
• Donate: Disseminate the results and contribute

This is not a waterfall process. It is an interactive iterative framework for innovation. Shneiderman's book focuses on the need to develop software to support this framework. The participants in the functional test tool workshop focused on the need to develop testing software to support those developing software to support this framework. And in doing so, we exhibited this framework in action -- without even identifying the framework. (I read about the framework on the plane home from the workshop.)

Gathering people that are interested in and working on solutions together accelerates the collection, creation, and donation. I expect great things to come from this gathering.

My thanks and appreciation go to the Agile Alliance for sponsoring this workshop; and to Ron Jeffries, Elizabeth Hendrickson, and Jennitta Andrea for organizing it.

Let's keep the innovation ball rolling and build "new computing" tools.

---

Links

• Next Generation Functional Testing, Geoffrey Wiseman
• Envisioning the Next Generation of Functional Testing Tools, Jennitta Andrea
• Functional Test Tools: the Next Generation, Elizabeth Hendrickson
• Portal Transparency, Ward Cunningham

Things We Know

I find it at work. I find it in online forums. I find it in books. I find it in papers. I find it in blogs. I find it at conferences.

I hear it from experts. I hear it from freshers. I hear it from friends. I hear it from managers. I sometimes even hear it come out of my own mouth.

It influences testers. It influences developers. It influences managers that influence testers and developers. It impacts customers.

It wastes time. It wastes money. It frustrates developers. It confuses executives. It demeans testers. It decreases quality in the name of improvement.

It permeates the practice of developing and testing software.



What is this ubiquitous it?




It is testing folklore.

It ain’t so much the things we don’t know that gets us in trouble. It’s the things we know that ain’t so.
- Artemus Ward

Here are some examples I pulled off the top of my head:

• There are best practices
• Tool vendors know those best practices
• The right tools make good testing
• Testers are the enemies of developers
• Automated unit testing is the only testing we need
• Written requirements are needed for testing
• It is possible to document unambiguous requirements
• Repeatability is maturity
• Tests can be completely designed and scripted before execution
• Testing is simple if guided by the right process
• Quality can be tested into a product
• Good manual testing can be replaced by automation
• Automation is only good for regression testing
• Test case counts are a good measure of test status
• All web pages should load in under 6 seconds
• Testers need to have development skills
• Good testing can be pre-scripted to be executed by anyone that can follow directions
• Boundaries are easy to identify
• Most bugs occur at boundaries
• Testing is easily outsourced to unintelligent people
• Testing is easily outsourced to tools
• Increased testing effort improves quality

What folklore do you encounter?

It is time to unlearn those things we know that ain't so. Challenge the folklore. Ask questions.

• Who says so?
• How do they know?
• What are they missing?
• Does it apply to my context?
• Does it make sense?

Maybe its time to call Mythbusters.

For Better or For Worse

I just stumbled across a 1986 quote from then-president of now defunct Ashton-Tate. I think it has application to automation in software testing.

"A computer will not make a good manager out of a bad manager.

It makes a good manager better faster and a bad manager worse faster."

-Ed Esber

All software is automation. Therefore, all software testing involves some level of automation.

Automation has potential to do good or bad faster. And, faster is not necessarily better.

Let's be smart about how and what we automate.

Woodpeckers, Pinatas, and Dead Horses

Here's some short blurbs of a few things I took away from CAST sessions.

From Lee Copeland's keynote address:

• "It's nonsensical to talk about automated tests as if they were automated human testing."

• Write or speak about something you're knowledgeable and passionate about.

• Combine things from multiple disciplines.

From Harry Robinson's keynote address:

• Weinberg's Second Law: If Builders Built Buildings The Way Programmers Write Programs, Then The First Woodpecker That Came Along Would Destroy Civilization.

From Esther Derby's keynote:

• To successfully coach someone, they must want to be coached and want to be coached by you.

From James Bach's tutorial:

• Pinata Heuristic: Keep beating at it until the candy comes out. ... and stop once the candy drops.

unless ...

• Dead Horse Heuristic: You may be beating a dead horse.

yet beware ...

• If it is a pinata, don't stop beating at it until the candy drops; but if it is a dead horse, your beating is bringing no value. It can be a challenge to determine if its a pinata or a dead horse.

From Antti Kervinen's presentation:

• Separate automation models into high level (behavior) and low level (behavior implementation) components to reuse test models on a variety of platforms and configurations.

More from James Bach's tutorial:

• Testing does not break software. Testing dispels illusions.

• Rational Unified Process is none of the three. (attributed to Jerry Weinberg)

From the tester exhibition:

• Testing what can't be fixed or controlled may be of little value. Some things may not be worth testing.

• There is great value in the diversity of approaches and skills on a test team.

• It may be possible to beat a dead horse and test (and analyze) too much. Sometimes we should just stop testing and act on the information we have.
  

From Doug Hoffman's tutorial:

• Record and playback automation can be very useful for testing for the same behavior with many configurations. And, once the script stops finding errors: throw it out.

From Keith Stobie's keynote:

• Reduce the paths though your system to improve quality. Fewer features may be better.

• Free web sites often have higher quality than subscription sites. This is because it is easy to measure the cost of downtime on ad-supported systems.

From David Gilbert's session:

• People expect hurricanes to blow around and change path. We should expect the same with software development projects. (David has some interesting ideas about forecasting in software development.)

• Numbers tell a story only in context. You must understand the story behind the numbers.

One more from James:

• Keep Notes!

What did you take away from CAST?

Exploratory Scripted Automated Manual Testing

Exploratory testing is often said to be the opposite of scripted testing. Automated testing is often said to be the opposite of manual testing. Instead of selecting one or the other, I find it helpful to look at these as opposite ends of spectra. I believe good testing contains components spread throughout multiple spectra.

I think we get into trouble when we apply these labels to “testing” as a whole. I believe all software testing has some aspects that are exploratory, some scripted, some automated*, and some manual. Hopefully all are driven by sapience.

Here are some things I remember about one of the best testing projects in my experience:

• Organizational testing processes were well documented
• Requirements were well documented
• We had enforceable quantitative requirements
• Tests were well planned
• Test plans were well documented
• Test cases were well defined and traceable to requirements
• Tests were well scripted
• Automation was an integral part of the testing

All of the above look like attributes of scripted, plan everything first, testing. We did a lot of planning and scripting for this testing. However, the following things are also true for this testing project:

• Process and documentation requirements were adaptable to the context
• Test "script" execution was mostly exploratory and mostly manual
• Bugs were only reported and officially logged if they were not fixed quickly
• Many requirements were qualitative
• Automation was “driven” by human testers


This scripted exploratory automated manual testing worked very well in its context. It was both the most scripted and most exploratory testing I can think of at the moment: it was the best testing.

Exploration does not mean leaving the map at home. Exploration is not the same as wandering. Testing was well planned while being exploratory in nature and practice. Human beings were involved at every step of the test planning and execution. Introducing automation reduced each iteration’s the test execution time from weeks to days without removing the human tester.

This testing was exploratory. It was dynamic. It was also well
scripted. The combination of static and dynamic components was central to the success of the project.

Instead of thinking of performing exploratory or scripted or automated or manual tests, find the balance of the four to create testing that works best in each situation.


* If you think no aspect of your testing is automated, try testing the software without a computer. And now that I wrote that, I am thinking of software tests that may not require a computer or software. It all depends on context. :)

Crapper Automation

Mechanization best serves mediocrity.
- Frank Lloyd Wright

Automation in public restrooms is becoming commonplace. The restrooms at my workplace have recently been remodeled and automated. The lights turn on and off automatically. The toilets flush automatically. Soap dispenses automatically. Faucets turn on and off automatically. Paper towels dispense automatically.

All of the above appear to be good candidates for automation. Automation in the restroom is supposed to improve cleanliness, reduce maintenance, and cut supply costs. However, I am not certain that this is true in implementation. I have found the following bugs in our newly automated water closets.

• I have to walk about ten feet into the restroom in the dark before the lights come on.


• It has been reported that if one spends too much time seated on the throne, the lights will turn off leaving the occupant to finish their business in the dark.


• The paper towel dispensers spit out towels when someone walks by the dispenser.


• The faucets come on when I stick my hand under the neighboring automatic soap dispenser but the water flow stops by the the time I get my hand to the water. I then have to remove my hands from the sink and put them back under the faucet.


• If I'm not quick enough, the soap dispensers drop the soap into the sink instead of on my hands.


• The water temperature is often too hot or too cold -- and there's no way to adjust it.


• Sometimes the soap dispenser drops soap on my clean hands as I remove them from the sink; requiring that I wash again.


• The paper towel dispenser spits out either too much or two little towel. One sheet is not enough to dry my hands but two is more towel than needed.


• Using two too-large towels contributes to trash receptacle overflow.
  


• It is difficult to retrieve items accidentally dropped in the sink without engaging the soap and water. I hope no one ever drops a cell phone or PDA in a sink.

None of these automated tasks are particularly sapient processes. However, design and implementation flaws in the automation have helped create new problems. Automation that was designed to save time and money can end up costing more if it is not properly implemented. A single mistake can be perpetually amplified by automation.

The first rule of any technology used in a business is that automation applied to an efficient operation will magnify the efficiency. The second is that automation applied to an inefficient operation will magnify the inefficiency.
- Bill Gates

The privy automation reminds me of the time I installed a home automation system in a former home. I installed an automatic doorbell that would ring when people stepped on my porch. I put some lights on motion detectors so they'd automatically go on and off as people entered and exited rooms. I set other motion detectors to turn off manually-engaged lights some time after the last detected motion. Outside lights were configured to not come on between sunrise and sunset. My bedroom lights came on just before the alarm clock went off. I could turn lights on and off with a remote from anywhere in the house. I had a single button I could press when leaving the house that turned off all the lights.

My home automation required a great deal of tweaking after it was installed. I discovered that I needed motion detectors at both ends of the hallway to make the lights come on before I got half way through the hall. I learned that turning off lights in the bathrooms and bedrooms based on motion was a bad idea. I had to reconfigure outdoor motion detectors to prevent the lights and doorbell from going on when cats walked across the front yard. It took a few months to get everything configured as I liked it. By the time I had everything tweaked right, my electric bills had dropped by 20%. I was happy with this automation and it paid for itself in a couple years -- plus I had fun playing with the gadgets.

Whether we are automating toilets, towel dispensers, household lights, or software testing tasks: good design, implementation, and testing are a necessity. In my experience, intelligent application is more important than the tools.

I'd like to hear your automation success and failure stories. Please share them here.

Sapience In The Age Of Automation

Society has unwittingly fallen into a machine-centered orientation to life, one that emphasizes the needs of technology over those of people, thereby forcing people into a supporting role, one for which we are most unsuited. Worse, the machine-centered viewpoint compares people to machines and finds us wanting, incapable of precise, repetitive, accurate actions. Although this is a natural comparison, and one that pervades society, it is also a most inappropriate view of people. It emphasizes tasks and activities that we should not be performing and ignores our primary skills and attributes -- activities that are done poorly, if at all, by machines. When we take the machine-centered point of view, we judge things on artificial, mechanical merits. The result is continuing estrangement between humans and machines, continuing and growing frustration with technology and with the pace ans stress of a technologically centered life.

- Donald A. Norman
Things That Make Us Smart:
Defending Human Attributes in the Age of the Machine

In his book, Things That Made Us Smart, Donald Norman argues that the technology that has made us smarter by allowing us to manage the artifacts of cognition needs to be made to conform to people instead of the more common practice of people conforming to technology.

Society often heralds the benefits of machines and overlooks the wonders of the human mind. This seems to be especially true in software development and testing. Many people talk about automating all tests with the assumption that the traits of the machine are better than those of human testers.

In any activity done by a human, there is the human aspect (not practically mechanizable), a physical aspect involving translation or transformation of matter and energy (mechanizable in principle), and a problem-solving aspect (sometimes transformed by mechanization, sometimes not affected).
- James Bach, Sapient Processes

If we limit our definition and practice of testing to the strengths of the machine, we are not really testing: we are overlooking the insight, intelligence, and wisdom that sets us human beings apart from the machines we create.

(noun) sapience ability to apply knowledge or experience or understanding or common sense and insight

In his Blog post Sapient Processes, James Bach applies the term sapience to processes that require skilled human beings. Sapient processes cannot be fully automated. Testing is a sapient process.

Automation can be a useful tool to help us manage the artifacts of cognition during testing. However, no automation on its own is sapient. Therefore, no automation on its own is testing.

Don't anthropomorphise the machines. Doing so requires stepping down to the level of machines.

The future masters of technology will have to be light-hearted and intelligent. The machine easily masters the grim and the dumb.
- Marshall McLuhan

Bad Messages

I regularly spend a great deal of time tracking down the root cause of software errors -- both on and off the job. Much of the investigation effort could have been avoided if I were not presented with incomplete or incorrect error messages. The text of error messages appears to be commonly overlooked by software developers and testers.

One of my early test automation development tasks was to fix the error messages in a test tool. The tool was used to validate the structure of data exchanged between computer systems. In nearly all cases, this tool displayed an error dialog window stating that the data was not valid. This tool gave the user (a tester) no information about what was wrong with the data. The lack of detail in these messages required that the user manually examine the data -- bit by bit -- with a protocol analyzer. This turned investigation of errors reported by an automation tool into a tedious manual task. The tool encountered a problem but did not report the source of the problem. I reviewed the code and added more detail to the error messages in this test tool. I also added logging of the data exchange in a format that made sense to human beings that could be reviewed without a protocol analyzer. The improved error messages and human-readable logging greatly decreased the troubleshooting and investigation time.

I am amazed at the poor error messages in test automation tools. Test automation tools and frameworks are likely to encounter errors in the software under test. I believe that this requires better error reporting and handling than many other tools. I am also disappointed in how difficult some of the automation tools make it to create scripts that run unattended.

One tool I use regularly often stops executing tests and displays a message like the following.

It took a great deal of effort to figure out how to coax the tool into returning error codes that could be handled in code instead of prompting a user during an automated test execution. The tool vendor's support personnel didn't seem to understand why it was a problem that I could not code their tool to handle an error without human intervention at run time. They kept telling me to fix the application under test to make the test tool's error dialog go away.

I also recently uninstalled a different test tool that had a horrible uninstall interface. It displayed a window with an uninstall progress bar that didn't move. After several minutes, the application displayed an "Install Complete" window underneath the progress window. The progress bar began to slowly move only after I clicked the "Finish" button on the buried completion window. Then it prompted me with something like the following.

Do I want to erase all files? What's the context? Of course, I don't want to erase all my files. I wonder how much the maker of this software spends on support calls about this message.

I used to receive numerous inquires from new computer users about the Windows "program has performed an illegal operation" error message. Many users thought this meant that they did something illegal and the police would soon be knocking on their door. People familiar with computers understand that this error message is about the program's interaction with the operating system and hardware; however this error message is misleading. This error message is designed for developers, not users. Good error messages tell the user what they need to know about the problem and what to do about it. Good error messages explain the problem but do not overwhelm the user with information that is useless to the intended user. This means that different kinds of applications require different kinds of error messages.

If you write software, please provide your users with accurate messages tailored to the user -- not developers. If you create test tools or automation frameworks, provide testers with information that is useful in determining what happened. Please.

People who write framework software should spend more time on useful error messages that show people why the error occurred and give a clue as to how to fix it.
- Eric M. Burke, We need better error messages

Want to practice creating better error messages? Try the Error Message Generator.