#004 Test Thursday: the A/B Test Hypothesis Before the Test

"Let’s test the button in green." Sounds data-driven. Usually it’s gut feeling with a testing tool behind it. You change a colour, switch the test on, check it after three days and wonder why the supposed win never shows up in revenue.

An A/B test without a hypothesis is guessing with a coat of numbers. Episode 004 of the Conversion Hacker Podcast is about the question that has to come before every test: what exactly do you expect, and how would you know you were wrong?

What makes a hypothesis

A good test hypothesis answers three questions in one sentence. It says why you expect a change. It says which metric should move. And it says from which measured value the change counts as proven.

The template is short and fits in one sentence: "Because we believe that [reason], we expect [metric] to change by [amount] within [period], measured by [KPI]."

An example from practice: "Because we believe shoppers are surprised by the shipping costs in the cart, we expect checkout completion to improve by 4 percentage points within eight weeks, measured as orders per checkout start."

Someone else can check the work against this sentence. They can redo the maths, work out how many orders are needed and say at the end whether the expectation came true. That’s exactly what a sentence like "test the button in green" lacks.

Why a bad result is worth more

The reflex after a losing test is to bin the hypothesis and build the next test. In doing so you throw away the information you’ve just paid for.

A test that loses has disproved your assumption. That clears away a question that would otherwise have stayed open for the next six months. At that moment the information is worth more than the revenue the test didn’t bring.

There’s one exception, and it belongs here: a test without a hypothesis that loses is worthless. It disproves nothing, because it never claimed anything. You can see the difference in the documentation. A test with a hypothesis gets one line: "We expected X, we measured Y, so we’re doing Z." A test without a hypothesis gets nothing at all.

We record this in a ledger file, one line per test, with the date, the hypothesis and the final decision. After twenty tests you know which of your assumptions have already been disproved. You don’t have to test them again, and you save half the hypothesis work a team without this list reinvents every year.

The three most expensive setup mistakes

The first mistake is a missing baseline. Without clean tracking you won’t know at the end whether conversion changed or the measuring point shifted. It’s the most expensive mistake because it makes the test worthless while it runs, and you often only notice at the end.

The second mistake is the multiple testing effect. You test 20 variants a month and look at the 20 numbers. One of them will look good by chance.

That isn’t a test, it’s a lottery with 20 tickets, and the hit rate looks like success even though it’s pure chance. The only fix is a fixed evaluation regime set before the start: one test, one decision, one time window.

The third mistake is reading the result too early. Three days are almost never enough, unless the effect is huge. Read the result after three days and you find a random value and call it a trend. Again, one detail has to be set beforehand: when you’ll read the result, and the threshold at which the test counts.

The fourth point is a decision you have to make explicitly: segmentation. Before the start, decide whether you look at all devices together or separately. In the checkout, separating them is often more interesting, because an effect behaves differently on a phone than at a desk.

The hypothesis in practice

A calculation: a shop with 60,000 sessions a month and a 1.50 percent conversion rate makes 900 orders at €180, so €162,000 a month. The hypothesis: "Because we believe a summary at the start of the checkout settles the cost question, we expect 5 percent more conversion within eight weeks."

A test with two equal arms needs around 3,500 orders per arm to detect a 5 percent difference. At 60,000 sessions a month that’s about eight months of run time. If there isn’t enough time, the honest answer is a smaller hypothesis. Then you test for two percent, and that gets its own test.

The point is that you know this in advance. The hypothesis becomes a calculation, the calculation a time window, the time window a decision. You can see from this chain that the hypothesis also sets the test duration. That’s the practical reason to write it down first.

How this fits into your day-to-day

The hypothesis is the cheapest part of the testing process. It costs one sentence and saves weeks. Skip it and testing gets more expensive, because every result says nothing and the next idea starts from scratch again.

The hypothesis comes first. Everything after it is execution.