Math paradoxes

Glossary / Minimum detectable effect (MDE)

Glossary · minimum detectable effect

How long does a test of +2% take?

An A/B test seems to be a matter of a few weeks: launch it, look, decide. But every test sees only effects above a certain bar. Smaller ones it mostly misses, however real they are. The bar depends on traffic, conversion rate and test length, and it is called the minimum detectable effect, or MDE. On a small site it can be so high that a modest improvement goes unseen for years.

In plain wordspictures, analogies, small numbers

“In plain words” is hidden on all puzzles.

A shop with a thousand visitors

An online shop gets 1,000 visitors a day, and 3% of them buy. A new checkout page really is better: with it 3.06% of visitors will buy, which is 2% more. For a big shop that is real money. The owner splits traffic evenly between the old and the new page and wants a test that finds such an effect with 80% probability at the usual 5% significance level.

How long does such a test have to run?

A test length calculator

The line shows the smallest lift the test will notice depending on how many days it runs (the day scale is logarithmic). The dashed line is the lift you want to notice. Where the curve comes down to the dashed line, the test is long enough.

MDE after this many daysthe lift you want

The test will take–
Visitors per version–
MDE in 4 weeks–

The bar below which a test is blind

The simplest explanation, slightly simplified. In any test the number of purchases jitters at random: today 16 people bought on the old page, tomorrow 13, though the page is the same. A real difference has to be clearly bigger than this jitter, or it gets lost in it. The more visitors, the less the share of buyers jitters, and the lower the bar. The size of this jitter is measured by the standard error.

The minimum detectable effect is the smallest real effect that a test of a given size will find with a probability chosen in advance, usually 80%. It is worked out before launch: from the conversion rate, the traffic and the test length. Effects smaller than the MDE the test sometimes finds too, but mostly misses.

The treacherous part is how slowly the bar comes down. A test twice as long does not make it half as high: it drops by only 29%. To notice an effect half the size, you need four times as many visitors. That is why small improvements are so expensive to check.

A test that is short of days

The computer runs the same test from the calculator above 200 times: the new page really is better by exactly the lift you chose. Each dot is one test. Change the length and see how often the test finds the real improvement.

new page significantly better: effect found"no difference""old page significantly better"

MDE for this length–
The test found the effect–
By the formula–

When the real effect is far below the MDE, the test finds it hardly more often than it would find a "winner" between two identical pages. Such a test cannot tell a working change from a useless one: whatever it shows, you have learned almost nothing.

Rigoroustables, charts, formulas

“Rigorous” is hidden on all puzzles.

The formula

For two versions with conversion rates p₁ and p₂, the number of visitors needed per version at a 5% significance level (two-sided) and an 80% chance of finding the effect is:

n ≈ (1,96 + 0,84)² · [p₁(1 − p₁) + p₂(1 − p₂)] / (p₂ − p₁)² ≈ 16 · p(1 − p) / (p₂ − p₁)²

Here 1.96 corresponds to the 5% significance level, 0.84 to 80% power, and 2 · 2.8² ≈ 15.7 is rounded to 16. The MDE is the same formula solved the other way round: n is given, and we look for the smallest difference. For conversion it is handy to count not visitors but purchases per version, c = n·p. Then the relative MDE is about 2.8 · √(2(1 − p) / c), and for small conversion rates it is simpler still: MDE ≈ 4 / √c.

Purchases per versionMDE, exact (3% conversion)Rule 4 / √c

A hundred purchases per version, and the test notices only effects of around +40%. To get down to +4% takes 10,000 purchases per version. What counts is the number of purchases: a site converting at 1% has to gather about three times as many visitors as a site at 3% to have the same MDE.

The MDE is not a cut-off, it is 80%

The MDE is often pictured as a line: anything bigger the test will see, anything smaller it won't. In fact the chance of finding an effect, which is statistical power, changes smoothly. An effect equal to the MDE is found in 80% of cases, not always. Here is how the chance depends on what share of the MDE the real effect is (normal approximation; it hardly depends on conversion or traffic):

Real effectThe test finds it

Another surprise: the significance bar lies below the MDE. A test planned for an MDE of +10% already calls an observed lift of about +7% significant, because 1.96 / 2.8 ≈ 0.7. So a "significant +7%" in such a test is no wonder but an ordinary thing. The wonder would be a test with a real +10% effect always showing at least +10%: half the time it shows less.

What to do about it

  • Pick the MDE from money, not hopeAsk from what lift the change pays for itself. That is the smallest effect worth catching. Work out how many days it needs before you launch.
  • Longer is slowDoubling a test's length lowers the MDE by only 29%. If it needs years, no "let's give it another week" will help.
  • Little traffic, bolder changesA small site can test a new price, offer or the whole checkout, where a +30% effect is realistic. Small tweaks it sees only as noise.
  • A more frequent event, a lower MDEAdding to the cart happens more often than buying, so a test on it notices smaller effects. But it measures something else, and its link to revenue must be checked separately.
  • Each extra version raises the MDEFour versions instead of two: each gets half the traffic, and the MDE grows √2 ≈ 1.41 times. And random "winners" turn up more often too.
  • Read "not significant" through the MDEA test with no result says only that an effect bigger than the MDE is probably not there. About effects smaller than the MDE it says nothing.
Deepergeneralisations, theory, history

“Deeper” is hidden on all puzzles.

The rule of sixteen

In 1992 the statistician Robert Lehr of the pharmaceutical company Berlex published a short paper in Statistics in Medicine titled "Sixteen S-squared over D-squared". He proposed estimating the size of each group as n ≈ 16·s²/d², where s² is the variance of the measurement and d is the difference to be noticed. The rule works for exactly the usual setting: 80% power and a two-sided 5% significance level. For conversion, s² = p(1 − p), which gives the formula above.

Where the 16 comes from: the difference of the means of two groups of n wobbles with a standard error of s·√(2/n). For a real difference d to be found in 80% of cases, it has to lie 1.96 + 0.84 = 2.8 such errors away from zero: d = 2.8 · s · √(2/n). Hence n = 2 · 2.8² · s²/d² ≈ 15.7 · s²/d². The square in the denominator is what makes small effects so expensive: make d ten times smaller and n grows a hundredfold.

All A/B test calculators use this same logic: they ask for the conversion rate and the MDE and answer with a number of visitors. If the test is driven by the traffic you have, it helps to count the other way round, as in the calculator above: as many days as you have, that is the MDE you get. Sometimes the honest answer is "this test makes no sense on our traffic", and that too is a useful decision made before launch.

Where it shows up

More on “Data and experiments”

All terms →