A marketer shows the same page as "A" and as "B". The true conversion rate of both is 5%. The dashboard counts conversions and says how far ahead the leading page is. How often will one of two identical pages beat the other by 20% or more? Lock in your answer, then check it in a game, on a simulation of 20,000 tests and in the explanations: what a distribution, a standard error and a p-value are, and how many visitors you really need.
A marketer shows the same page as "A" and as "B", both converting at 5%. In 36% of such tests one page "beats" the other by 20% or more. With 1,000 visitors per page that is plain noise: about 50 conversions scatter by ±7. Below are a game, a simulation of 20,000 tests and the explanations: distributions, the standard error, p-values and how many visitors you really need.
An A/A test looks like an ordinary A/B test, but both variants are the same. It is run to check the tool and to see how much "difference" the noise alone produces.
A landing page with a "Buy" button. The testing tool shows it to half the visitors as "A" and to the other half as "B". Nothing differs: not the copy, not the colour, not the price.
Each visitor buys with probability 5%, independently of the others and of the "version" they saw. Each variant gets 1,000 people, so on average there will be 50 purchases each.
At the end the dashboard shows the conversions and how many percent the leader beats the other page by. For example: "A 52, B 61, B +17%". We ask about a "win" of 20% or more, for example 50 against 60.
In what share of such A/A tests will one of the two identical pages get at least 20% more conversions than the other?
"Rarely, a few percent: the pages are identical"
In fact it happens in 35.8% of tests, more than one in three. 50 conversions is a small sample, even when a thousand people visit. A gap of 10 purchases is routine here.
Two identical pages, a 5% conversion rate, 1,000 visitors each. In what share of tests will one of them get at least 20% more conversions?
The exact answer and a breakdown of your estimate will appear here.
Hard to decide? Open the hints one at a time: the easy one first, then stronger ones.
Count purchases, not visitors. How many conversions does each page get on average? Would you trust a poll of that many people?
The number of purchases from 1,000 visitors at 5% wobbles by about √50 ≈ 7 either way. The difference of two such independent numbers wobbles √2 times more.
A 20% "win" at 50 purchases means a gap of about 10 purchases. Compare that with the typical spread of the gap. How rare is a deviation of one typical deviation?
Here are 10 A/B test dashboards. Behind each one the computer secretly flipped a coin: either the pages are identical (both 5%), or B really is 10% better (5.5%). Mark which ones you think are a real lift and which are noise, then press "Check". Then raise the number of visitors.
"Chance B is better" is computed the way many tools show it: Φ(z), where z is the difference in conversion rates divided by its standard error. A star ★ means 95% or more, the threshold at which a tool usually says "significant". In tests where the pages are identical this happens in one test in twenty.
The computer runs 20,000 tests in two worlds. In the grey world the pages are identical. In the dark one B really is 10% better. For each test it records the lift a dashboard would show and stacks those lifts into bars. Two "bells" appear. Drag the slider: the more visitors, the narrower the bells and the easier it is to tell them apart.
Solid bars beyond the dashed lines: the grey ones are false winners, the dark ones on the right are the real effect the test caught (power). Lift here is the difference in conversions B − A divided by the expected 5% (at 1,000 visitors that is 50 purchases). A dashboard divides by A's actual conversions; the picture barely changes. A false winner is an A/A test in which either page beat the other past the p < 0.05 threshold (the usual two-sided z-test). Exact values come from adding up binomial probabilities over all pairs of outcomes.
The dark line is the lift the marketer sees as visitors pile up. The grey lines are 20 more tests just like it. The band holds 95% of the values noise can produce and narrows like 1/√n. Early in a test the dashboard shows huge "wins" that do not exist.
From a coin to the sample-size formula. The first layer has no formulas, the second has tables and a calculator, the third covers theory and history.
“In plain words” is hidden on all puzzles.
Flip two ordinary coins 100 times each. Each lands heads about 50 times, yet one beats the other by 10 heads or more 17.9% of the time. Nobody would call one coin "better".
A page visitor is a coin too, only "heads" comes up 5% of the time. 1,000 visitors give the same 50 purchases, but the scatter is bigger: ±7 instead of ±5, because rare events wobble more (√50 ≈ 7). Two identical pages differ by 10 purchases or more in 32.9% of tests.
The gap between two pages wobbles √2 ≈ 1.4 times more than each page does: both carry noise. So ±20% is not a rare event but one "typical deviation". This is a simplification; the exact calculation in round 1 gives 35.8%.
"We had a thousand visitors, that is a lot" sounds convincing. But only the people who bought carry the information, and there are 50 of them. Nobody calls a poll of 50 people precise, yet a test on 50 conversions often is.
That is why a page converting at 1% needs about five times the traffic of a 5% page to detect the same relative lift. The table in the "Strict" layer shows it.
After the first day each variant has a handful of purchases, and 3 against 6 reads "+100%". The more people, the narrower the band of noise on the trumpet chart in round 3. It narrows like 1/√n: four times the visitors give a band half as wide.
| Visitors per variant | Noise in the gap (95%) |
|---|---|
| 250 | ±76% |
| 1,000 | ±38% |
| 4,000 | ±19% |
| 16,000 | ±9.6% |
The p-value answers the question: if the pages were identical, how often would noise produce a gap this big or bigger? p < 0.05 means "less than 1 time in 20". It is a statement about noise, not about your page.
Whether B really is better also depends on how often your ideas work at all. If only one in ten does, a noticeable share of even the "significant" winners are fake. A calculator is in round 5.
First decide on the smallest lift that matters to you. Then work out how many visitors you need to detect it in 8 tests out of 10 (round 5 has a calculator). Then wait until the test has that many and only then look at the result. If you do not have that much traffic, test bolder changes: a +20% lift needs four times fewer people than +10%.
“Rigorous” is hidden on all puzzles.
The width of the bell from round 3 is measured by the standard error. For one page's conversion rate it is √(p(1 − p)/n), for the difference between two pages √(2p(1 − p)/n). About 95% of noise results fall within ±1.96 standard errors. Here are those limits for p = 5%.
| n | SE of gap, pp | ±1,96·SE, pp | as lift |
|---|
pp means percentage points: 5.0% → 5.5% is a difference of 0.5 pp, a lift of +10%.
z = the difference in conversion rates divided by its standard error (using the pooled rate of both groups). The p-value is two-sided, from the normal curve. The 95% confidence interval for the difference is ±1.96 standard errors. With small numbers the normal curve is a little optimistic, so Fisher's exact test is shown next to it.
| Base conversion | lift +5% | +10% | +20% | +50% |
|---|
A statistical power of 80% means: if the effect really is this big, the test detects it in 8 cases out of 10. Half the lift needs four times the people, because the sample size grows like 1/δ², where δ is the absolute difference in conversion rates. The quick rule n ≈ 16·p(1 − p)/δ² gives 30,400 for 5% and +10% instead of the exact 31,234. For small lifts it is off by a few percent; for +50% it falls short by up to a fifth.
| Test says "effect" | Test says "nothing" | |
|---|---|---|
| B really is better | True winnershare = power, 1 − β | Missed ideatype II error, β |
| Pages identical | False winnertype I error, α = 5% | Honest "nothing"1 − α = 95% |
The p < 0.05 threshold controls only the bottom row: among A/A tests 5% are false winners, whatever the number of visitors. The top row depends on the sample size. With 1,000 visitors per variant a +10% effect is detected in only 7.2% of tests.
The number of conversions in a variant with n visitors is binomial. The standard error of a rate and of the difference of two rates.
Our puzzle: p = 5%, n = 1,000. The spread of the gap as a lift, that is divided by 5%.
The z-test with the pooled rate p̄ of both groups, and the two-sided p-value; the 95% confidence interval for the difference.
The sample size per variant for a difference δ = p₂ − p₁, two-sided α and power 1 − β.
The first row explains where the bell comes from: the number of conversions is a sum of a thousand "coins". The second gives the width of the noise in our puzzle: one standard deviation of the gap is almost a 20% lift. The third is the standard check the calculator above does. The last shows why an effect four times smaller costs 16 times the visitors: δ is squared in the denominator.
“Deeper” is hidden on all puzzles.
The bars are the exact binomial probabilities of how many of 1,000 visitors buy at a 5% rate. The line is a normal distribution with the same mean of 50 and spread of 6.9. The central limit theorem says that a sum of many small independent random bits has roughly this distribution, whatever each bit looked like. That is why every bell on this page has the same shape; only the width changes.
Ronald Fisher, in "Statistical Methods for Research Workers" (1925), proposed looking at the p-value and called the 1-in-20 limit a convenient one for deciding whether a deviation counts as significant. For him it was a measure of evidence in a single experiment.
Jerzy Neyman and Egon Pearson, in papers from 1928 to 1933, took another view: a rule that makes decisions over and over. They introduced type I and type II errors and the power of a test. A modern A/B test mixes both views: the p-value from Fisher and the sample-size calculation from Neyman and Pearson.
B really is 10% better. We keep only the tests where B won significantly and look at the lift they reported. With 1,000 visitors per variant a test is significant only when noise happened to add a lot: on average such tests show a dashboard lift of +60%. At 5,000 it is +24.1%, and only from about 30,000 does a significant result stop lying about the size of the effect. That is why the "winner" of a small test almost always delivers less than it promised once rolled out to everyone.
Revenue per visitor is noisier than conversion: on top of the chance of "bought or not" comes the chance of the order size. If the average order is 50 but orders are spread out, as they usually are (lognormal, σ = 1.2), the variance of revenue per visitor is 4.4 times larger, and the same precision needs 4.4 times the visitors. A single order worth a few thousand can make a "winner" on its own.
First the maths (how many visitors and how many days), then the logic (what "significant" and "not significant" mean), then practice (how to check that the tool itself is not lying).
The sample size grows like 1/δ². Doubling the expected lift cuts the test fourfold. So sites with modest traffic should test not the shade of a button but a new offer, a price or the page structure: changes that could give +20–50%.
Run a test in whole weeks, even if the sample fills up sooner: people buy differently on weekends and weekdays. And checking every day and stopping at the first "95%" is not allowed; that is a trap of its own: peeking at an A/B test.
Picture 1,000 tested ideas, of which one in ten works. A test with 80% power finds 80 true winners, while 5% of the 900 empty ideas, that is 45, "win" by chance. Of the 125 significant results only 64% are real. At 20% power the share drops to 31%. It is the same mechanism as in the false positive paradox: the rare real thing drowns among the frequent false ones.
Absence of evidence is not evidence of absence. With 1,000 visitors per variant a +10% effect stays "not significant" in 92.8% of tests. You cannot conclude "we tested it and nothing changed" from such a test: it could barely see anything.
Before serious tests it pays to run a few A/A tests. If clearly more than 5% of them come out "significant", the tool counts wrongly: for example, one visitor lands in both groups, or bots go to only one.
The second check costs nothing: did the traffic really split the way it was planned? 50,000 against 48,500 on a 50/50 split looks like a detail, but by chance it happens less than once in half a million. This is a sample ratio mismatch: visitors of one variant get lost somewhere, for example because of slow loading. The results of such a test cannot be trusted, however nice the lift.
Ship's surgeon James Lind, aboard HMS Salisbury, split 12 sailors with scurvy into six pairs and gave each pair a different remedy: cider, vinegar, seawater, two oranges and a lemon a day, and others. Only those who ate citrus recovered quickly. Lind described the trial in "A Treatise of the Scurvy" in 1753.
Ronald Fisher wrote in "Statistical Methods for Research Workers" (1925) that P = 0.05, or 1 in 20, corresponds to 1.96, "or nearly 2" standard deviations, and that it is convenient to take this point as the limit of significance. Fisher himself later advised choosing the significance level to suit the circumstances.
In 2008 Barack Obama's campaign tested the button and the image on its sign-up page. A "Learn More" button and a family photo raised conversion from 8.26% to 11.6%. Dan Siroker, the campaign's director of analytics, estimated that this brought about 2.9 million extra email addresses and roughly 60 million dollars in donations.
In 2012 a Microsoft employee suggested changing how Bing displayed ad headlines. The idea was ranked low priority, and for more than six months nobody touched it. When an engineer finally ran an A/B test, revenue rose 12%. Ron Kohavi and Stefan Thomke in Harvard Business Review (2017) put it at more than 100 million dollars a year in the US alone.
Ron Kohavi and colleagues reported in "Online Experimentation at Microsoft" (2009) that only about a third of the ideas tested improved the metric they were designed to improve. The rest turned out neutral or harmful. That is exactly why a small test that "finds" a winner every time should raise suspicion.
Designer Douglas Bowman wrote a farewell post when he left Google in March 2009. He said a team that could not choose between two shades of blue tested 41 shades in between to see which performed better. Bowman gave it as an example of a culture where data had to prove every design decision.
Share your own explanation, an objection or a variant of the puzzle. You need a GitHub account to comment.
Is “Advanced” the right level for this puzzle?
Thank you! This helps set the levels right.
I keep adding new puzzles, new ways to explain them and write-ups of the discussions they start. Leave your email and I'll write when something new is out. No spam, one-click unsubscribe.
The sign-up form will appear here soon.