Metrics from where the sun don't shine

Posted by Ben Simo

Numbers don't lie, people do.

Posted by Ben Simo

“despite its mathematical base, 
statistics is as much an art as it is a science.
… Often the statistician must choose among methods,
a subjective process,
and find the one
 that he will use to represent the facts.
This suggests giving statistical material …
a very sharp second look
 before accepting any of them. …
But arbitrarily rejecting statistical methods 
makes no sense either.

That is like refusing to read 
because writers sometimes
use words to hide facts and relationships
rather than to reveal them.”

- Darrell Huff,
How To Lie With Statistics, 1954


“No doubt 
some graphics do distort
 the underlying data,
making it hard for the viewer 
to learn the truth.
But data graphics are no different 
from words in this regard,
for any means of communication 
can be used to deceive.”

- Edward Tufte,
The Visual Display of Quantitative Information

Talkin' 'bout test in a differrent light

Posted by Ben Simo

Pullin' out my big black book
Cause when I need a word defined that's where I look
So I move to the L's quick, fast, in a hurry
Threw on my specs, thought my vision was blurry
I look again but to my dismay
It was black and white with no room for grey
Ya see, a big "V" stood beyond my word
And yo that's when it hit me, that luv is a verb.

Words come easy but they don't mean much
When the words they're sayin' we can put trust in
We're talkin' 'bout love in a different light
And if we all learn to love it would be just right.

- DC Talk


The DC Talk song "Luv is a Verb" points out that love is something to be acted out. Real love is action, not just words and feelings. Love is expressed through action.

Like love, the word test is both a noun and a verb. Also like love, test requires action. Even the noun definitions for test describe action.

test

noun

• trying something out to find out about it
• any standardized procedure for measuring sensitivity or memory or intelligence or aptitude or personality, etc.
• a set of questions or exercises evaluating skill or knowledge
• the act of undergoing testing
• the act of testing something

verb

• put to the test, as for its quality, or give experimental use to
• test or examine for the presence of disease or infection
• examine someone's knowledge of something
• show a certain characteristic when tested
• achieve a certain score or rating on a test
• determine the presence or properties of (a substance)
• undergo a test
  

from Princeton WordNet

While I don't think we'll find anyone that argues that test is not a verb, people involved in software development seem to use it primarily as a noun. I have nothing against the many things we create that we call tests. Our test cases, test code, test charters, and whatever test things we create can be useful tools -- but they are not the test.


Let's think about test in a different light.

Several years ago, Shrini Kulkarni challenged me in questioning whether there can be such a thing as an automated test. I don't think I disagreed with Shrini. I've not been one to trust testing to machines, but I've been a fan of automation throughout my testing career. I've automated many testing tasks, but not believed I can automate the testing itself.

Earlier this year, Michael Bolton told me of a distinction he was thinking about between checking and testing. While I had no disagreement with this distinction, I wasn't thrilled with the terms. I wanted something more descriptive. I thought Michael was making a distinction I had been trying to make: a distinction between validation and investigation. I've since come to understand that Michael is making a slightly different distinction. Michael has recently written a series of blog posts better describing the Checking vs. Testing distinction. Michael has limited the scope of checking to observations and decisions rules that can be executed without sapience -- without a brain-engaged human. If something requires human sapience, it is testing, not checking.

Yesterday, an insightful tester, Lanette Cream, made a nice attempt at defining test on her blog. In her latest revision, she defines test as follows.

A test is
an action
which produces discoveries
that can be used to evaluate product quality.

I like that this definition identifies action, discovery, and evaluation as being core to testing. However, I'm thinking of pushing, or rather constraining, this just a bit further.

What if we were to say that the evaluation is the action and discovery is the goal?

A test would then be the sapient part of validation or investigation -- the thinking and learning that cannot be automated. All those other things we do to test are really support activities that help us evaluate.

Test is not a document. Test is not code. Test is not executing a program. Test is not applying a procedural decision rule. Test is not anything that can be done by a machine. Test is the act of evaluating. Test requires sapience.

Test is thinking and learning that leads to discovery. We may test by evaluating existing data. We may test by running experiments that produce new data. We may take the output of automated checks to test. We may provide what we learn as input to coding new automated checks. The test is the action we perform in our minds.

This may come across as nitpicking vocabulary. That's not my intent. My goal is not to limit anyone's definition of test, but rather to shed a different light on what I believe sets testing apart from checking, and gives both checking and testing value.

If a check fails in a forest and no one is around to hear it, does it make a sound?

The true value of our checking and testing is in the mind of a sapient tester. What value is there in all the things we call checks and tests without a tester (whatever their role or title) evaluating information and learning?

Test is sapient evaluation that leads to discovery

.

I'm not quite comfortable with this. I want the emphasis to be on the sapient activity; and not generating and collecting data to support the thinking without ignoring that it is a necessary part of testing.

Regardless of where we shine the light or draw lines, let's keep in mind that test is a verb.

What do you think? Testing of my half-baked ideas is welcome and appreciated.

Comprehensive Understanding?

Posted by Ben Simo

"Anyone who feels
that he
can precisely define
the boundaries of his profession
either
possesses a skill of little importance
or
is incredibly naive."

- William F. Sharpe,
The Economics of Computers, 1969

Marginal value & cost of software

Posted by Ben Simo

* Image from the Economics of Computers, by William F. Sharpe.

Cost of additional copies approaches zero ONLY IF no maintenance or customer support costs are associated with sales of additional copies.

If your quality stinks, expect that supposedly 99% profit copy you sold me to increase your costs.

Quality improvement that reduces maintenance and support costs can have great value!

If scripted tests can't find bugs...

Posted by Ben Simo

Shrini Kulkarni (@shrinik) tweeted that a friend told him if exploratory testing finds bugs not found by scripted testing, it may be due to insufficient or incorrect test planning and review.

Perhaps there aren't enough test cases. Perhaps testing techniques weren't applied properly. Perhaps the review of tests was done wrong.

However, perhaps -- just perhaps -- people are fallible and can't completely understand or work through the complexity of their customer's problems and the technical implementation of a solution with finite knowledge, finances, and time.

If it is reasonable to expect testers to design enough tests to exercise near-infinite paths with near-infinite data variations in a software system, then I think it should be reasonable to expect software designers and coders to anticipate everything that could go wrong and prevent all threats to the value of the software they produce -- all with finite finances and time.

If this were reasonable, then it would be reasonable to skip testing altogether.

In the real world, it is just as unreasonable to expect perfection from test case designers as it is to expect perfection from all the other people involved in developing software. Is it not?

One of the things exploratory testing helps avoid is locking in our level of ignorance that exists at the start. An explorer can use information gained as they explore to help guide where they go and what they do next.

So you think you've achieved maturity?

Posted by Ben Simo

Some people say that Ada may be the last major high level language that will ever be developed, since automatic program generation techniques may be available in the not too distant future. Thus it seems fitting that the last major programming language be named in honor of the first female prorammer.

- Richard Wiener and Richard Sincovec,

Programming in Ada, 1983

Arrogance combined with foolish optimism? :)

Freedom & Responsibility Culture

Posted by Ben Simo

Cool. Based on this slide deck, it appears that Netflix has a good understanding of developing and sustaining a corporate culture of Freedom & Responsibility.

Culture

View more presentations from reed2001.

F.A.I.L.U.R.E.

Posted by Ben Simo

F.A.I.L.U.R.E.

A mnemonic for testing error handling & reporting.

F.A.I.L.U.R.E.

View more presentations from Ben Simo.

Is There A Problem Here?

Posted by Ben Simo

Over the years, I have collected a number of examples of software failures in the wild -- some I've encountered myself, some were shared by others. I've had intentions to create a blog for sharing these software failures, and new ones as they are discovered, with hope that software designers, developers, and testers can discuss and learn from them. I have finally launched that blog. It is titled Is There A Problem Here?

I invite you to visit the blog and contribute at http://IsThereAProblemHere.com.

I'm helping you. I'm helping you.

Posted by Ben Simo

A few months ago, I enlisted my 11 year old son to help me with some work around the house. After a short while, he was doing something other than what I had asked him to do.

I told him, "You're not helping me."

"But I am helping you.", he replied.

"No you're not."

"I'm helping you. I'm helping you.", he shot back. He was frustrated. He really thought he was helping me; and I was putting down his work. I was frustrated too. From my view, his helping was creating more work for me. I did not feel helped.

Then it hit me. I've heard this argument before -- from software testers.

I've seen testers, and test managers, attempt to justify their work by telling team members and stakeholders "I'm helping you. I'm helping you." We QA and tester people develop metrics and reports to help us demonstrate how helpful we are. We talk about our quality assurance and testing processes. We talk about all the test cases we develop and execute. We like to show off our test automation that spits out impressive color-coded results.

However, we still encounter unhappy team members and stakeholders. We develop adversarial relationships with developers. We have to explain ourselves to project leads that question the value of our testing. We hear people tell us we're not helping and we keep saying "I'm helping you. I'm helping you."

Maybe, just like my son, we're not giving our stakeholders what they need. Maybe we aren't really helping. So instead of shooting back the "I'm helping you." line, we can stop and listen. Find out what our stakeholders want from us. Listen and ask clarifying questions to better understand how we can help.

I'm not advocating that we just give in and do whatever we're asked without defending our positions. However, we can be willing to adjust our positions to better serve our stakeholders. (Joining an overly optimistic rush to release poor quality software usually doesn't serve them.) If there is disagreement, work to resolve it. Sometimes we may need to educate others on our areas of expertise. Yet we testers also need to respect others' roles and expertise. Listen and learn.

So, the next time you feel like screaming "I'm helping you. I'm helping you.", try to better understand how you can help before turning up your defenses.

Serve your stakeholders.