Why does a weak test exaggerate its wins?

The winner's paradox

The computer runs the same A/B test 2,000 times on a button that really is 10% better. Each bar shows how many tests saw that "lift" on the dashboard. The magenta ones came out statistically significant: these are the ones that get presented at the meeting as a "win". Change the number of visitors per variant.

The bar is higher than the effect

The simplest explanation, slightly simplified. For a test to call a difference significant, it has to be bigger than the usual noise. With 1,000 visitors per variant the noise is large, and the significance bar sits at around "+40%". A real +10% falls far short of it. Only tests where noise happened to add another 30 points can jump the bar. So every "winner" here is a lucky one, and its lift is always exaggerated.

The share of tests that notice a real effect of a given size is called statistical power, or the power of a test. A power of 20% means: if the idea really works exactly this well, the test will recognise it in one case out of five.

• Silence is not emptinessAt 20% power, the test will say "no difference" four times out of five, even though the idea works. A weak test with no result doesn't prove the change is useless: it simply saw nothing.
• A win is not a sizeWhen a weak test does find an effect, trust the direction, not the number. The next, bigger test will almost always show less. That is not "the effect wore off", it is an honest estimate instead of a lucky one.
• Power is computed before the testIt depends on the effect you want to notice, the base conversion rate and the number of visitors. Work it out in advance, and you'll know whether the test makes sense at all.

What power depends on

The line shows the power of the test against the number of visitors per variant (the scale is logarithmic: each tick is ten times the previous one). The usual rule is to plan a test so that the power is at least 80%. See how far to the right that point moves when the effect is small or the conversion rate is low.

The square in the denominator makes small effects expensive. An effect half the size needs four times as many visitors. So when traffic is low, it is better to test bold changes rather than the shade of a button.

A table for the 5% → 5.5% button

An exact calculation with the binomial distribution, no simulation. The "bar" is the smallest lift at which the test, with a 5% conversion rate in variant A, already calls B significantly better. The last column is the average dashboard lift among significant wins for B.

While the power is below 50%, the bar is higher than the effect itself, and every win necessarily exaggerates. When the power reaches 80%, the winners show +11.4% on average, already close to the truth.

What to do about it

• First the effect, then the trafficDecide what the smallest effect is that still matters to you, and work out how many visitors 80% takes. If that is half a year of traffic, the test in this form makes no sense.
• Little traffic, bolder changesA +30% effect is about eight times cheaper to notice than +10%: at a 5% conversion rate it takes about 3,800 visitors per variant instead of 31,000. On a small site, test a new offer or price, not the colour of a button.
• More events, more powerWhat counts is the number of conversions, not visitors. A click on a button happens more often than a purchase, so a test on clicks is more powerful. But it also measures something else.
• "Not significant" ≠ "doesn't work"Before you close an idea, check what power the test had. At 20% it would miss a working idea four times out of five too.
• A weak test's win gets recheckedRepeat the test or run it to the planned size. Expect the lift to shrink, and plan the budget around the smaller number.

Type M and type S errors

In 2014 the statisticians Andrew Gelman and John Carlin proposed looking beyond power to two other errors. A type M (magnitude) error is how many times, on average, a significant result exaggerates the real effect. A type S (sign) error is the share of significant results that show the effect with the wrong sign. The line below is the theory for the normal approximation, the magenta dot is your 2,000 tests from the first simulation.

Under the normal approximation the exaggeration depends on power alone: at 80% it is 1.12 times, at 50% 1.41, at 20% 2.26, at 10% 3.71. And at 6% power, one significant result in five has the wrong sign: the test "proves" that the better version is worse. So power is not only the chance of finding an effect, but also a measure of how far a finding can be trusted.

Where 80% came from

The term "power" and the error of the second kind were introduced by Jerzy Neyman and Egon Pearson in a 1933 paper on the most efficient tests. An error of the first kind is seeing an effect that isn't there; of the second kind, missing a real one. Power equals one minus the probability of an error of the second kind.

In 1962 the psychologist Jacob Cohen checked the papers in the Journal of Abnormal and Social Psychology for 1960 and found that for medium-sized effects the power of the studies averaged about 0.48: almost a coin toss. In his book on power analysis (second edition, 1988) he proposed 80% as a convention for when there is no better basis. The logic: with a 5% significance level and a 20% miss rate, a false alarm is treated as roughly four times worse than a missed effect.

Half a century changed little. In 2013 Katherine Button and colleagues in Nature Reviews Neuroscience assessed 730 studies from 49 meta-analyses in neuroscience: the median power was 21%. And back in 1995 the statisticians Douglas Altman and Martin Bland gave the phenomenon a name in the BMJ worth remembering after every "non-significant" test: "Absence of evidence is not evidence of absence".