The test is honest: a big sample, one look at the result, a "significance" threshold of p < 0.05. Overall there is no difference. But then the result is sliced into 20 countries and each is checked on its own. What is the chance that at least one country shows a "significant" difference? Lock in your answer, then try to find a winner on the dashboard yourself.
Each check has a 5% chance of a false alarm. Twenty checks give a 1 − 0.95²⁰ = 64% chance that at least one country "lights up", though email B is no different from A. The more slices, metrics and variants, the surer a chance "winner" appears. Below: a game, a simulation and ways to protect yourself.
This is the third page in a series on A/B tests. The first, the A/A test, shows how noise makes "winners" in a single test. The second, peeking, shows what happens when the same test is checked every day. Here the test is single and honest, but its result gets looked at from many sides.
A marketer tests a new email template B against the old A. Each subscriber randomly gets one of the two. We know what the marketer doesn't: B is exactly as good as A. There is no real difference anywhere, in any country.
The sample is big, its size fixed in advance, and the result is read once, at the end. The overall result honestly says: no difference.
Then the analyst splits the result into 20 countries. The countries' audiences do not overlap, so the 20 comparisons are independent. Each is checked with the usual test at p < 0.05, the same threshold behind the dashboard's "significant, 95% confidence".
What is the chance that in at least one country B "significantly" wins or loses?
"5%: we set the test to 95% confidence"
In fact 64%. The 95% confidence protects one check. Each of the 20 countries gets its own 5% chance of a false alarm, and the chance that all 20 stay quiet is 0.95²⁰ ≈ 36%.
B is no different from A. 20 independent countries, each tested at p < 0.05. What is the chance that at least one country shows a "significant" result?
A breakdown of your estimate and 1,000 simulated campaigns will appear here.
Hard to decide? Open the hints one at a time: the easy one first, then stronger ones.
A p < 0.05 threshold means that even with no difference, 1 check in 20 says "significant" on average. How many such false alarms should you expect from 20 checks?
It is easier to count the opposite: what is the chance that none of the 20 countries lights up? For one country it is 0.95.
The countries are independent, so the chances of staying quiet multiply: 0.95 × 0.95 × … twenty times. Subtract the result from 1.
Here is the dashboard of a real-looking campaign: 40,000 subscribers, half got A, half got B. The overall result is flat, and the boss wants to hear where the new email worked. Slice the data any way you like: country, device, new or loyal subscribers, send day, signup source, and three metrics. Each new slice adds a row to the report. Your task: find where B "significantly" wins.
Lift is relative to A: +20% means the metric is 1.2 times higher in B. The bar spans lifts from −60% to +60%, stretched near zero. Slices with fewer than 10 events in total are not tested: there is too little data.
The computer runs thousands of A/A campaigns: B always equals A. First we count how the chance of a false winner grows with the number of slices k. The lines are the exact formula, the dots the simulation. Then we look at the p-values themselves, and finally at the case where the metrics are linked to each other.
The Bonferroni correction divides the threshold by the number of checks. The Benjamini–Hochberg method is more lenient when real effects exist, but when there are none at all it also keeps the chance of a false winner at 5%. Both are explained in round 4.
Opens, clicks and purchases are linked: someone who did not open the email will not click. Metrics that move together more often stay quiet together and light up together, so the chance of at least one false winner is below 64%. But even at a strong correlation of 0.7 it is almost six times the promised 5%. Model: 20 test statistics sharing a correlation ρ.
Every slice gives the noise one more chance to "win". First in plain words, then tables, corrections and formulas, finally the theory.
“In plain words” is hidden on all puzzles.
Imagine that each check is a ticket that gives a fake "prize" with a 1 in 20 chance. One ticket almost never wins. Twenty tickets give exactly one prize on average. The chance that none wins is 0.95 × 0.95 × … twenty times, which is 0.95²⁰ ≈ 36%. So at least one false winner appears 64% of the time.
This is a simplification: the tickets here are independent, like the countries in our puzzle. Overlapping slices and linked metrics behave a little differently, as round 3 showed, but the core is the same.
Tickets are not bought by countries only. 5 countries × 2 devices × 3 metrics × 2 periods ("first week" and "whole month") is already 60 looks. Were they independent, the chance of at least one false winner would reach 95%. Variants count too: an A/B/C/D test compares three emails with the control, not one.
Each such look costs one click in the report. So people make hundreds of them without noticing that each buys another ticket.
As soon as a segment lights up, an explanation turns up on its own: "Poles like minimalism", "loyal subscribers appreciate the new design", "people read more carefully at weekends". The story sounds convincing because it was picked to fit the result. Had Spain lit up, there would be a story about Spain.
That is how false insights are born: first the noise, then the story, then a decision for the whole list.
Before launch, write down one primary metric (say, clicks) and a short list of segments that really matter for the decision. Check the primary metric at the usual threshold. Either check the segments with a correction for how many there are, or treat them as hypotheses for the next test.
A finding you did not plan for is not forbidden. But it is a question, not yet a conclusion: check it on fresh data. In the round 2 game, the "What was really there?" button runs such a check.
“Rigorous” is hidden on all puzzles.
| Checks k | At least one false | False on average | Bonferroni threshold |
|---|---|---|---|
| 1 | 5.0% | 0.05 | 0.05 |
| 5 | 22.6% | 0.25 | 0.01 |
| 10 | 40.1% | 0.5 | 0.005 |
| 20 | 64.2% | 1 | 0.0025 |
| 50 | 92.3% | 2.5 | 0.001 |
| 100 | 99.4% | 5 | 0.0005 |
Chance of at least one: 1 − 0.95ᵏ. Average count: k × 0.05, it grows exactly in proportion to k. With the Bonferroni correction the chance of at least one false winner stays at 4.9–5.0% for any k.
The simplest correction divides the threshold by the number of checks: for 20 countries only p < 0.05 / 20 = 0.0025 counts as "significant". Then the chance of at least one false winner is back to about 5%. But a strict threshold also misses real effects. An effect the ordinary test detects with probability 80% (a statistical power of 80%) is detected only 41% of the time after the correction. To get back to 80%, each country's sample must grow 1.9 times.
The same goes for the confidence interval on the dashboard: 95% means that out of 20 intervals one misses the true lift on average. For all 20 countries together to cover the truth at least 95% of the time, each interval must be widened to 99.75%, about 1.5 times: 3.02 instead of 1.96 standard errors each way.
Bonferroni guards against any false finding at all. That is harsh when there are hundreds of checks. In 1995 Yoav Benjamini and Yosef Hochberg proposed something else: control the share of false ones among the findings (the false discovery rate, FDR) so that on average it stays at most 5%. The rule: sort the p-values from smallest, and find the last one lying under the sloping line i × 0.05 / k. It and all smaller ones count as findings.
| Method | Findings | Of them false |
|---|
k independent checks, each with a false-alarm chance α. The chance of at least one (the family-wise error rate, FWER) and the average number of false ones V.
The Bonferroni correction works for dependent checks too, since the chance of a union of events is at most the sum of the chances (Boole's inequality). The Šidák correction is exact for independent ones.
The Benjamini–Hochberg method: p₍₁₎ ≤ … ≤ p₍ₖ₎ are the sorted p-values, q the target share of false ones among findings. For independent tests the share of false ones averages π₀q, where π₀ is the share of tests with no effect.
The price of Bonferroni for k = 20: an effect giving 80% power at α = 0.05 is δ = 1.960 + 0.842 = 2.802 standard errors. A threshold of 0.0025 means z = 3.023.
The first row gives 64%. The second shows why Bonferroni can be used without knowing how the checks are linked: it is an upper bound. The third describes the Benjamini–Hochberg method. The last computes the price of the correction: the chance to detect a real effect falls from 80% to 41%, and getting back to 80% takes a 1.9 times larger sample.
“Deeper” is hidden on all puzzles.
The round 3 histogram is flat for a reason. A p-value is a "tail" share: how rare a gap like this is among identical emails. If there is no effect, the gap itself comes from that very distribution, so the event "p ≤ t" happens with probability exactly t. So p < 0.05 occurs in 5% of tests, p < 0.5 in half, and p between 0.40 and 0.45 as often as between 0 and 0.05.
That is why multiple testing needs no assumption about the data: k tests with no effect give k uniform random numbers, and sooner or later the smallest of them falls below 0.05.
In 2013 Andrew Gelman and Eric Loken gave this name to the situation where an analyst runs one test but picks it after seeing the data. Drop the "outliers" or not, count from day one or day three, compare clicks or opens, merge Poland with Czechia or not. Each choice is reasonable, but with different data the choice would have been different.
So one test was run, but many were possible, and the p-value no longer means "1 time in 20". The cure is the same: a plan before the data and a check on fresh data.
A test of four new emails against one control A gives 4 comparisons, but not independent ones: they all compare with the same A. If A came out weak by chance, all the newcomers look better together. With equal groups the correlation between the comparisons is 0.5. Then the chance of at least one false winner is 15.6% instead of 18.5% for independent ones, still three times 5%.
For this case there is an exact correction by Charles Dunnett (1955): a threshold of |z| > 2.44, that is p < 0.0146 for each comparison. It is a little more lenient than Bonferroni (p < 0.0125), because it accounts for the comparisons being linked.
First the maths of hunting for a winner, then the logic: what happens when only the "significant" gets published. Finally, practice for a marketer.
The analyst tries independent slices one after another and stops at the first "significant" one. Each try succeeds with probability α. The number of the first successful try has a geometric distribution: 1/α tries on average, that is 20 at α = 0.05. Half of the hunts end within 14 tries.
A stricter threshold does not stop the hunt, it only makes it longer: at α = 0.01 it takes 100 tries on average. A dashboard with a thousand possible slices gives plenty. The round 2 game has 1,944 slices, and they overlap, so the hunt there is a little different, but it ends the same way.
In the xkcd comic "Significant" (xkcd.com/882) scientists check whether jelly beans cause acne. No link. Then they test 20 colours one by one, and the green ones come out at p < 0.05. The newspaper headline: "Green jelly beans linked to acne! Only 5% chance of coincidence!" That is exactly our puzzle.
Now imagine 100 labs studying an effect that does not exist. About 5 get a "significant" result and publish it, the rest put their data in a drawer. Readers see only 5 papers, all confirming the effect. What is more, the published effects are always big: 2.34 standard errors from zero on average, because a smaller one cannot pass the threshold. This is the "winner's curse" from the first page of the series, multiplied by the number of labs.
The same happens in a company when the weekly meeting shows only the tests that "worked".
The adjusted p = min(1, k × p). It is below 0.05 only if the finding would pass the 0.05 / k threshold.
Peeking at a test every day is the same multiple testing, only in time: each evening buys a new ticket. That is the second page of the series. And how many people one honest test needs to show anything is worked out on the first.
The large ISIS-2 trial (The Lancet, 1988, over 17,000 heart attack patients) showed that aspirin reduces deaths. The authors deliberately added a split by star sign: for people born under Gemini or Libra aspirin seemed not to help (deaths +9%, not significant), while for everyone else it cut them by 28%. It was a warning about how misleading subgroups can be.
Craig Bennett and colleagues put a dead Atlantic salmon in an fMRI scanner and showed it photos of people in social situations. Without a correction for multiple testing a few voxels of its brain "responded". A scan has tens of thousands of voxels, each tested separately. With the correction the activity vanished. In 2012 the work won the Ig Nobel Prize in neuroscience.
xkcd comic No. 882, "Significant", by Randall Munroe came out on 6 April 2011. Scientists test 20 colours of jelly beans at p < 0.05, and the green ones are "linked to acne". The comic became a classic illustration of multiple testing in statistics textbooks and lectures.
That is what John Ioannidis called his 2005 paper in PLoS Medicine. He showed that with small samples, small effects, many tested hypotheses and flexible analysis, most "significant" results in a field can be false. According to Wikipedia, as of 2020 it was the most viewed article in the history of the publisher PLOS, with more than 3 million views.
The Open Science Collaboration repeated 100 studies from three psychology journals (Science, 2015). In the originals 97% of results were significant, in the repeats only 36%. The repeat effects were on average half as big. It is one of the best-known episodes of the replication crisis.
Yoav Benjamini and Yosef Hochberg published their method for controlling the false discovery rate in the Journal of the Royal Statistical Society, Series B (1995). Today it is one of the most cited statistics papers: in 2005 it was named among the 25 most cited. The method became standard in genetics, where thousands of genes are tested at once.
Share your own explanation, an objection or a variant of the puzzle. You need a GitHub account to comment.
Is “Advanced” the right level for this puzzle?
Thank you! This helps set the levels right.
I keep adding new puzzles, new ways to explain them and write-ups of the discussions they start. Leave your email and I'll write when something new is out. No spam, one-click unsubscribe.
The sign-up form will appear here soon.