Glossary / Minimum detectable effect (MDE)
An A/B test seems to be a matter of a few weeks: launch it, look, decide. But every test sees only effects above a certain bar. Smaller ones it mostly misses, however real they are. The bar depends on traffic, conversion rate and test length, and it is called the minimum detectable effect, or MDE. On a small site it can be so high that a modest improvement goes unseen for years.
“In plain words” is hidden on all puzzles.
An online shop gets 1,000 visitors a day, and 3% of them buy. A new checkout page really is better: with it 3.06% of visitors will buy, which is 2% more. For a big shop that is real money. The owner splits traffic evenly between the old and the new page and wants a test that finds such an effect with 80% probability at the usual 5% significance level.
How long does such a test have to run?
About seven years: 2,563 days.
The difference between 3% and 3.06% is six purchases per 10,000 visitors. The random noise in the number of purchases over a few weeks is far larger. For six extra purchases per ten thousand to stand out from the noise, it takes about 1.28 million visitors per page. In two weeks this shop can reliably see only an effect of +29% or more.
The line shows the smallest lift the test will notice depending on how many days it runs (the day scale is logarithmic). The dashed line is the lift you want to notice. Where the curve comes down to the dashed line, the test is long enough.
MDE after this many daysthe lift you want
The simplest explanation, slightly simplified. In any test the number of purchases jitters at random: today 16 people bought on the old page, tomorrow 13, though the page is the same. A real difference has to be clearly bigger than this jitter, or it gets lost in it. The more visitors, the less the share of buyers jitters, and the lower the bar. The size of this jitter is measured by the standard error.
The minimum detectable effect is the smallest real effect that a test of a given size will find with a probability chosen in advance, usually 80%. It is worked out before launch: from the conversion rate, the traffic and the test length. Effects smaller than the MDE the test sometimes finds too, but mostly misses.
The treacherous part is how slowly the bar comes down. A test twice as long does not make it half as high: it drops by only 29%. To notice an effect half the size, you need four times as many visitors. That is why small improvements are so expensive to check.
The computer runs the same test from the calculator above 200 times: the new page really is better by exactly the lift you chose. Each dot is one test. Change the length and see how often the test finds the real improvement.
new page significantly better: effect found"no difference""old page significantly better"
When the real effect is far below the MDE, the test finds it hardly more often than it would find a "winner" between two identical pages. Such a test cannot tell a working change from a useless one: whatever it shows, you have learned almost nothing.
“Rigorous” is hidden on all puzzles.
For two versions with conversion rates p₁ and p₂, the number of visitors needed per version at a 5% significance level (two-sided) and an 80% chance of finding the effect is:
n ≈ (1,96 + 0,84)² · [p₁(1 − p₁) + p₂(1 − p₂)] / (p₂ − p₁)² ≈ 16 · p(1 − p) / (p₂ − p₁)²
Here 1.96 corresponds to the 5% significance level, 0.84 to 80% power, and 2 · 2.8² ≈ 15.7 is rounded to 16. The MDE is the same formula solved the other way round: n is given, and we look for the smallest difference. For conversion it is handy to count not visitors but purchases per version, c = n·p. Then the relative MDE is about 2.8 · √(2(1 − p) / c), and for small conversion rates it is simpler still: MDE ≈ 4 / √c.
| Purchases per version | MDE, exact (3% conversion) | Rule 4 / √c |
|---|
A hundred purchases per version, and the test notices only effects of around +40%. To get down to +4% takes 10,000 purchases per version. What counts is the number of purchases: a site converting at 1% has to gather about three times as many visitors as a site at 3% to have the same MDE.
The MDE is often pictured as a line: anything bigger the test will see, anything smaller it won't. In fact the chance of finding an effect, which is statistical power, changes smoothly. An effect equal to the MDE is found in 80% of cases, not always. Here is how the chance depends on what share of the MDE the real effect is (normal approximation; it hardly depends on conversion or traffic):
| Real effect | The test finds it |
|---|
Another surprise: the significance bar lies below the MDE. A test planned for an MDE of +10% already calls an observed lift of about +7% significant, because 1.96 / 2.8 ≈ 0.7. So a "significant +7%" in such a test is no wonder but an ordinary thing. The wonder would be a test with a real +10% effect always showing at least +10%: half the time it shows less.
“Deeper” is hidden on all puzzles.
In 1992 the statistician Robert Lehr of the pharmaceutical company Berlex published a short paper in Statistics in Medicine titled "Sixteen S-squared over D-squared". He proposed estimating the size of each group as n ≈ 16·s²/d², where s² is the variance of the measurement and d is the difference to be noticed. The rule works for exactly the usual setting: 80% power and a two-sided 5% significance level. For conversion, s² = p(1 − p), which gives the formula above.
Where the 16 comes from: the difference of the means of two groups of n wobbles with a standard error of s·√(2/n). For a real difference d to be found in 80% of cases, it has to lie 1.96 + 0.84 = 2.8 such errors away from zero: d = 2.8 · s · √(2/n). Hence n = 2 · 2.8² · s²/d² ≈ 15.7 · s²/d². The square in the denominator is what makes small effects so expensive: make d ten times smaller and n grows a hundredfold.
All A/B test calculators use this same logic: they ask for the conversion rate and the MDE and answer with a number of visitors. If the test is driven by the traffic you have, it helps to count the other way round, as in the calculator above: as many days as you have, that is the MDE you get. Sometimes the honest answer is "this test makes no sense on our traffic", and that too is a useful decision made before launch.
A sample some people get into more easily than others, so it describes everyone badly.
In 1936 a magazine polled millions of Americans and still picked the wrong election winner.
The more repetitions, the closer the share of an outcome gets to its true probability.
The share of heads creeps towards 50%, while the gap between heads and tails in coins usually grows.
A number from −1 to +1 showing how closely two quantities move together.
In summer both ice cream sales and drownings go up. Ice cream isn't to blame: the heat drives both.
How far an average or a rate computed from a sample usually misses the truth: σ divided by the square root of the sample size.
Halving the error takes four times as much data.
A range around a measured number, built by a method that covers the true value in a chosen share of samples, such as 95%.
Of twenty honest 95% intervals, on average one misses the truth, and none of them knows whether it is the one.
How often chance alone would produce a gap at least as big as yours if there were really no difference.
When there is no effect at all, p below 0.05 turns up as often as p between 0.50 and 0.55.
The number on Bayesian dashboards: how likely B is to be better than A, given the data and a starting assumption.
"97% chance" tells the truth only when every idea of yours changes something; if four in five are useless, B is better in only a third of such tests.
The probability that a test notices a real effect of a given size. A power of 80% means the test misses one working effect in five.
A weak A/B test that does find an effect almost always exaggerates it: a real +10% shows up on the dashboard as +24%.
A way to get the error of any number without a formula: redraw a sample of the same size from your own data with replacement many times and watch how the result jumps.
Each resample loses a third of your data and repeats the rest, yet the spread of such copies shows the real error.
A small random share of customers who for a long time see none of the shipped changes, so that comparing with them shows what all the changes gave together.
Twenty changes a year, each one winning its A/B test, almost +50% together, yet customers who saw none of them buy only about 7% less.