How many tests do you need?

Posted by Ben Simo

“Testing is potentially an infinite process.” – James Bach

Nearly 30 years ago, Glenford Myers provided a self-assessment test to software testers in his book The Art of Software Testing. Much of this book is outdated; however, his test is still regularly used to illustrate the complexity of testing software. He asked readers to write down the test cases required to adequately test the following program.

The program reads three integer values from a card. The three values are interpreted as representing the lengths of the sides of a triangle. The program prints a message that states whether the triangle is scalene, isosceles, or equilateral.

Glenford Myers then fills the next page of the book with a list of 14 questions that should be answered by test cases for this simple program. He then makes the point that more complex programs are exponentially more difficult to test.

Elizabeth Hendrickson recently created a modern version of the program (no punch cards are required) and challenged testers to create test cases and test her version. See "Testing Triangles: a Classic Exercise Updated for the Web" in her blog.


Last summer, James Bach offered a new challenge to handful of testers sitting around the table at a Denver eatery. This challenge involves testing an even simpler system. James asked how many times we need to press the button on the following system to test 85% of the possible results.

You are asked to test a black box with a single push button. Pressing the button spins an internal wheel that randomly stops at one of 100 possible positions. If any of the 100 possible results (otherwise unknown to the user) encounter a bug, the box will burst into flames. The system has no other output.

Have an answer? To help, let's consider a couple other systems.

How many times do you need to flip a coin to get 85% of the possible results? A coin has two possible results. The first flip will get you 50% of the possible results. Each additional flip will have a 50% chance of not getting the remaining result. Therefore, it is possible -- although unlikely -- that we will ever get a result of heads and a result of tails.

How many times do you need to roll a die to get 85% of the possible results? A die has six possible results. The firs roll will produce a new value. The chance of getting a new value will drop as each new value is encountered. As with the coin, it is possible that we will never encounter all the possible results.

• 1 value: 83% chance
• 2 values: 67% chance
• 3 values: 50% chance
• 4 values: 33% chance
• 5 values: 17% chance

Let's go back to James' challenge. How many time do you need to press the button to test 85% of the possible results? How about 95% of the results? And I will add the question: Is 100% test coverage feasible?

Click here to test a JavaScript implementation of this black box.

What if this black box had 200 possible values? What if there were 307.200 possible values?

Have an answer? Please share it.