Shipulski On Design

At first glance, it seems easy to run a good test, but nothing can be further from the truth.

The first step is to define the idea/concept you want to validate or invalidate.  The best way is to complete one of these two sentences: I want to learn that [type your idea here] is true. Or, I want to learn that [enter your idea here] is false.

Next, ask yourself this question: What information do I need to validate (or invalidate) [type your idea here]?  Write down the information you need.  In the engineering domain, this is straightforward: I need the temperature of this, the pressure of that, the force generated on part xyz or the time (in seconds) before the system catches fire.  But for people-related ideas, things aren’t so straightforward. Some things you may want to know are: how much will you pay for this new thing, how many will you buy, on a scale of 1 to 5 how much do you like it?

Now the tough part – how will you judge pass or fail?  What is the maximum acceptable temperature? What is the minimum pressure? What is the maximum force that can be tolerated? How many seconds must the system survive before catching fire?  And for people: What is the minimum price that can support a viable business? How many must they buy before the company can prosper? And if they like it at level 3, it’s a go. And here’s the most importance sentence of the entire post:

The decision criteria must be defined BEFORE running the test.

If you wait to define the go/no-go criteria until after you run the test and review the data, you’ll adjust the decision criteria so you make the decision you wanted to make before running the test. If you’re not going to define the decision criteria before running the test, don’t bother running the test and follow your gut.  Your decision will be a bad one, but at least you’ll save the time and money associated with the test.

And before running the test, define the test protocol.  Think recipe in a cookbook: a pinch of this, a quart of that, mix it together and bake at 350 degrees Fahrenheit for 40 minutes.  The best protocols are simple and clear and result in the same sequence of events regardless of who runs the test.  And make sure the measurement method is part of the protocol – use this thermocouple, use that pressure gauge, use the script to ask the questions about price and the number they’d buy.

And even with all this rigor, good judgement is still part of the equation.  But the judgment is limited to questions like: did we follow the protocol? Did the measurement system function properly? Do the initial assumptions still hold? Did anything change since we defined the learning objective and defined the test protocol?

To create formal learning objectives, to write well-defined test protocols and to formalize the decision criteria before running the test require rigor, discipline, time and money.  But, because the cost of making a bad decision is so high, the cost of running good tests is a bargain at twice the price.

Image credit – NASA Goddard Flight Center

Comments are closed.