Falsifiability Testing

Posted by Ben Simo

The best experiments deduce an effect from the hypothesis and then isolate it in the very context where it may be disproved.
- Michael Kaplan & Ellen Kaplan,
Chances Are: Adventures In Probability

In Focusing on Falsifiability, Stuart Thompson writes the following about a tester-friend's statement that testers can add value from the start of a project if they understand the project and its direction.

"With a clear understanding of what the software was actually trying to do, his team was able to provide useful feedback to the developers even within the first couple of release cycles."

When testers know the problem that software is trying to solve, they can first focus on the things that matter. They can test the assumptions. They can identify contexts in which the assumptions and the implementation can be disproved. Project managers and developers are usually focused on the solution. Testers can help identify the new problems created by proposed solutions and provide information to help the team determine if the new problems are not as bad as the ones they solve.

EACH SOLUTION IS THE SOURCE OF
THE NEXT PROBLEM
We never get rid of problems. Problems, solutions, and new problems weave an endless chain. The best we can hope for is that the problems we substitute are less troublesome than the ones we "solve".

- Donald C. Gause & Gerald M. Weinberg,
Are Your Lights On? How to Figure Out What the Problem REALLY Is

As Stuart points out, testers often have difficulty proving their worth when they prevent bad things from happening. It is usually in hindsight that we see where bypassed testing could have helped.

An example came to my mind as I read Stuart's post.

A change was made to an application. The testers knew about the problem that forced an update to the software. However, the testers we not told the details of the proposed solution. Shortly before the planned release, the testers discovered that the solution created new problems that were worse than the problem solved. In addition to creating new problems, the solution only worked in one of many likely contexts.

The developers told the testers that the solution was good because the "business" people approved it. The "business" people trusted that the developers knew how to implement the business need. It appears that no one specifically tested the assumptions. No one tried to put the solution in contexts that it may not work. Early information sharing with testers likely would have exposed the new problems created by the solution before time was spent coding, testing, and then re-engineering the solution.

It is ironic -- although common -- that a decision made in haste due to time pressure delayed the update to the software. Time spent on QA and testing early is usually going to save time and money.

Had someone focused on the falsifiability of the solution, the problems caused by the solution would have been prevented before a single line of code was written.

I have heard testers complain that no one invites them to be involved early in the process. I too have joined in that chorus. A colleague recently reminded a group of testers that sometimes we aren't included early because we don't ask. Yes, sometimes getting involved early is as easy as asking.

Ask to be involved early. Seek out ways to focus on falsifiability instead of nit-picking incomplete implementations, and developers are likely to invite you back.