How do I turn a vague idea into something I can test?

A hunch that you keep half-saying is hard to test because it is built from words that sound clear and commit you to nothing. The example below is invented, and every number in it is a count taken from the texts shown, so the numbers illustrate the method and say nothing about how often it works.

A vague note and what is wrong with it

The starting note reads: “Small bakeries might pay to hear when a regular customer has stopped coming in. The owners probably notice too late. I am not sure who I would sell it to.” It runs to thirty words and contains two statements that evidence could confirm or contradict, which are that owners would pay and that owners notice too late.

The note also contains five terms that a stranger would have to guess at, namely small, regular, stopped coming in, too late and pay. Every one of those hides a decision the writer has not made, such as how many shops count as small or how long a silence counts as stopped.

Running the note through a single compression pass in an online thinking lab produced the result shown below. The pass restated the note as a working claim and then asked for one observable change that would make it true, which points at the next job without doing it.

Step one: write the claim with its numbers and definitions

A claim says what would be observed if you are right, with every vague word replaced by something countable. Replacing the five open terms in the note gives this version.

Single-shop bakery owners cannot name the regulars who have bought nothing for three weeks, and one in five of them would pay $15 a month to be told.

The figures of three weeks, one in five and $15 are guesses, chosen so that they can be wrong. The claim runs to twenty-eight words and still contains two checkable statements, but only two open terms remain, which are regulars, since nobody has said how often a customer must visit, and cannot name, since nobody has said how many misses count.

The philosopher Karl Popper argued in the 1930s that a statement earns its place in science by forbidding some observations. His argument concerns scientific theories and gives no evidence that founders who apply it do better, but the same test is a useful filter for a hunch, because a claim that forbids nothing cannot be checked.

The chart below follows the count of open terms through all four stages of the example, with the table repeating the numbers.

Step two: find the assumption that would sink the claim

Two parts make up the claim, and they carry unequal risk. If owners can already name their lapsed regulars, nobody needs to be told and the price is beside the point, whereas if owners cannot, the price becomes worth asking about. The first part therefore sits underneath the second, and testing the price first would spend effort on a question that may have no object.

To find the assumption under your own claim, ask of each part what else must be true for it to hold, and keep asking until the answer is something you could check with a few people inside a week. Rank the candidates by how badly the idea suffers if the assumption is false and by how little it costs to look. The assumption to test first scores high on the first measure and cheap on the second.

Step three: turn that assumption into a question

Applied to the bakery claim, the assumption becomes the question of whether a single-shop bakery owner can name most of the regulars who have bought nothing for three weeks. That clause has two open terms, regulars and most, so it is still imprecise, but it points at one observation rather than at a whole business. The question stage aims the test and adds no precision of its own, which is why the count of open terms stays flat between the claim and the question.

A useful question has a subject you can find, an observation you can make and an outcome that does not depend on your mood. Questions that begin with “why” or “how might we” fail the last requirement, because almost any answer to them can be defended.

Step four: write the experiment and the result that would stop you

An experiment is the question plus a procedure and a decision rule written down before any result arrives. Writing the rule first matters because evidence read without one tends to be bent toward the answer you hoped for, a familiar habit that needs no statistic to explain. For the bakery claim the experiment is the following.

Ask ten single-shop bakery owners to name from memory the regulars who have bought nothing for three weeks, where a regular is a customer who bought in at least five of the previous six weeks. Check each list against the till or loyalty records, and count an owner as a success if the list covers at least three quarters of the customers the records show as lapsed. Seven or more successes ends the idea. Four to six successes calls for a repeat with ten more owners. Three or fewer successes leaves the claim standing and moves attention to the price.

This text has no open terms left, and it has eleven possible results, from zero successes to ten, each tied to one of three decisions. To see how the rule behaves, the chart below assumes that half of all owners could pass. Under that assumption each of the 1,024 possible pass-or-fail patterns across ten owners is equally likely, and the bars count how many patterns produce each number of passes.

Patterns with seven or more passes number 176 of the 1,024, about 17 per cent, so a world in which half of all owners could pass would still end the idea roughly one time in six. The repeat band holds 672 patterns, about 66 per cent. This is arithmetic under a stated assumption and is not a measurement from any real owner.

Choosing a test over a build

The table below compares three ways of putting the bakery claim to work, using reasoning about what each can reveal and no study of how founders choose.

What ten owners cannot tell you

Ten owners cannot estimate what share of all owners can name their lapsed regulars, so the check screens for a gross error and nothing finer. Owners who agree to help are probably more attentive than average, which is reasoning and not a measurement, and it would make a surviving claim somewhat more convincing and a refuted claim somewhat less so.

Whether one in five owners would pay $15 a month stays untouched by this test, and the same four steps apply to it with a different assumption on top. Take your own note, count its open terms, replace each one with something countable and write the stopping result before you contact anyone. What remains unknown for any idea is whether the guesses in your claim were the right guesses, and only the first round of results can say.