A marketer runs a 30-day test and opens the dashboard every evening. As soon as the tool shows "95% confidence", they stop the test and declare a winner. But pages A and B are identical. What is the chance that within 30 evenings one of them "wins" anyway? Lock in your answer, then check it with a game, a simulation and the explanations. At the end there is an honest answer to "what if I can't wait".
If you look at the dashboard every evening and stop at the first "95%", identical pages "win" in 28% of tests, not 5%. Each look gives the noise another chance to cross the line. The 95% only covers a single check planned in advance. Check it with the game and the simulation, and round 5 has seven honest ways to avoid waiting for months, each with its price.
This is the second page of a series on A/B tests. The first one, "Two identical pages", shows how noise creates "winners" on its own. Here it is the same noise, plus one habit: looking at the result every day.
Pages A and B are identical. Every visitor buys with probability 5% on either page. There is no real difference.
Each page gets 1,000 new visitors a day. The test is planned for 30 days, which is 30,000 per page.
Every evening the marketer opens the dashboard. The tool runs the standard test for two conversion rates and shows "confidence". The first evening it reaches 95% (p < 0.05), the test is stopped and the leading page is declared the winner, whether it is A or B.
What is the chance that within 30 evenings one of the identical pages is declared the winner?
"5%: that is exactly what 95% confidence means"
In fact it is 28%. The five percent apply to one look planned in advance. Here there are thirty looks, and each one gives the noise a new chance to cross the line.
Model: a two-sided z-test for two proportions with a pooled conversion estimate; the number of purchases is random (binomial). Either page can become the winner.
An A/A test, 30 evenings, a check every evening and a stop at the first "95%". In what share of such tests is a winner declared?
A breakdown of your estimate will appear here.
Hard to decide? Open the hints one at a time: the easy one first, then stronger ones.
If you buy one lottery ticket a day for a month, does your chance of winning stay the same as with one ticket?
On each evening alone the chance of a false "95%" is 5%. But the evenings are not independent: the data on day 20 are 95% the same as on day 19.
So the answer lies between 5% and 1 − 0.95³⁰ = 79%. Which end is it closer to, if late evenings add almost no new data?
The computer secretly decides: either the pages are identical, or B really is 10% better (5.5% instead of 5%). You don't know which. Step through the evenings and stop the test when you decide there is a winner. Then switch to the honest marketer, who looks only on day 30, and compare the scores.
| Visitors | Purchases | Conversion |
|---|
A "false winner" means you declared a winner though the pages were identical. "Real effect found" means B really was better and you declared B the winner.
We run thousands of A/A tests on the same 30,000 visitors per page and change only how often someone looks: once, twice, every evening, every hour or after every visitor. The line is the exact calculation, the dots are a simulation with random purchases.
Lock in your answer in round 1 first: these charts show it.
Each line shows how far B is behind or ahead of A, in units of noise (z). The ±1.96 bounds mean "95% confidence". A dot marks the evening a line first leaves the band: that is when the impatient marketer would stop the test.
First in plain words, then strictly: the random walk, tables and the bounds that honestly keep 5% with several looks. At the end some theory, including why Bayesian dashboards don't save you either.
“In plain words” is hidden on all puzzles.
Each evening the noise gets a new chance to cross the 95% line by accident. One look is one ticket with a 1-in-20 chance of a fake prize. Thirty looks are thirty tickets, and the chances pile up. This is simplified: the tickets are not independent, because today's data almost entirely repeat yesterday's. That is why it comes to 28%, not 1 − 0.95³⁰ = 79%.
The tool computes confidence as if tonight were the only check, agreed before the start. It does not know that you looked yesterday and will look tomorrow. If you set the date in advance and look once, identical pages "win" in 5% of tests, as promised. If you stop at the first "95%", you yourself pick the evening that is luckiest for the noise.
The lift of B over A in one A/A test as visitors accumulate. The grey band is the ordinary spread of noise (±1.96 standard errors). After day one ±38% surprises nobody; after 30 days the spread narrows to ±7%. An early "win" of +25% looks impressive, but it is within the noise. More about this trumpet on the first page of the series, "Two identical pages".
Picture a roulette player who leaves the casino the minute they are ahead. The game is fair, but they will often report "I left ahead". A test is the same: you watch a wandering curve and record the result exactly when it is at a peak. That is why false stops cluster at the start: about 59% of them happen in the first week, when there is the least data.
The same habit inflates real effects too: a test stopped at a peak shows a bigger lift than the true one.
“Rigorous” is hidden on all puzzles.
Let Sₖ be how many purchases B is ahead of A by evening k. Each day a new independent piece of noise is added, so Sₖ is a random walk. Its spread grows like √k, and the test divides by it: z = Sₖ / (σ√k). On any single evening |z| > 1.96 happens with probability 5%. But a walk that touches the curve ±1.96σ√k at least once in 30 steps is much more common.
| Evenly spaced looks | 1 | 2 | 5 | 10 | 30 | 100 |
|---|---|---|---|---|---|---|
| False winners | 5.0% | 8.3% | 14.2% | 19.3% | 28.0% | 37.4% |
Exact calculation for the normal approximation. The answer depends only on the number of evenly spaced looks, not on the traffic.
| Pair of evenings | Shared data | Correlation of z |
|---|---|---|
| 1st and 2nd | 50% | 0.71 |
| 10th and 11th | 91% | 0.95 |
| 29th and 30th | 97% | 0.98 |
| 1st and 30th | 3% | 0.18 |
The correlation between z on evening k and on evening m > k is √(k/m). Late in the test a new evening adds only a few percent of new data, so the new "ticket" is almost a copy of yesterday's. Early on the evenings differ a lot, and that is where most false winners are born.
| Day | 95% | Pocock | O'Brien–Fleming |
|---|---|---|---|
| 6 | 1.96 | 2.41 | 4.56 |
| 12 | 1.96 | 2.41 | 3.23 |
| 18 | 1.96 | 2.41 | 2.63 |
| 24 | 1.96 | 2.41 | 2.28 |
| 30 | 1.96 | 2.41 | 2.04 |
| False overall | 14.2% | 5.0% | 5.0% |
| Max sample | – | +23% | +3% |
The numbers are bounds for |z|. Pocock's 2.41 corresponds to 98.4% confidence at every look. "Max sample" is how many more visitors you must plan for to keep 80% power, compared with a single look.
Stuart Pocock (1977) proposed one stricter bound for all looks. Peter O'Brien and Thomas Fleming (1979) made the early bounds very strict and the last one almost ordinary: stopping on day 6 takes a huge gap, but on day 30 you lose almost nothing. Both schemes are set so that over all five looks a false winner appears in 5% of A/A tests.
Evening k: each page has nₖ = 1,000·k visitors, purchase shares p̂_A and p̂_B, pooled share p̄. The dashboard computes z and the confidence.
With no effect zₖ behaves like Sₖ/√k, where Sₖ is a sum of k independent standard normal steps. One look gives 5%. The chance of crossing at least once in K looks is computed step by step: the density of Sₖ inside the band is convolved with a normal step, and whatever leaves the band is set aside.
Bounds that keep 5% over K = 5 looks: Pocock uses one c for every k, O'Brien–Fleming uses a bound that falls like 1/√k.
We computed these probabilities by step-by-step integration on a fine grid and checked them with a simulation that draws exact binomial purchase counts: 400,000 A/A tests with the habit from the puzzle gave 28.1%, and the Pocock and O'Brien–Fleming bounds gave 5.0% and 5.1% over 200,000 tests each.
“Deeper” is hidden on all puzzles.
The law of the iterated logarithm (Aleksandr Khinchin, 1924) says that a random walk Sₙ reaches the height √(2n ln ln n) again and again. That height grows faster than 1.96·√n, so the walk will sooner or later cross any fixed bound for z with probability 1. It grows slowly: 30 looks give 28%, 100 looks 37%, hourly checks for a month about 50%. But there is no ceiling.
During the Second World War Abraham Wald invented the sequential probability ratio test (SPRT): after each observation, decide "enough" or "more". Its bounds are designed precisely for continuous peeking. A modern version, mSPRT, was described by Ramesh Johari, Pete Koomen, Leonid Pekelis and David Walsh in "Peeking at A/B Tests" (2017). It gives "always valid" p-values: you can look at them any time and false winners stay within 5%. The price is that for a fixed test size this approach is less sensitive than one planned look.
Many tools show not a p-value but a "chance that B beats A". It is often said you can look at it as much as you like. Let's check: A/A tests, a uniform Beta(1, 1) prior, a winner is declared when either page's chance reaches 95%.
| A/A tests, 95% bar | One look | Every evening |
|---|---|---|
| Either page "wins" | 10.1% | 46.7% |
| Only B "wins" | 5.0% | 23.8% |
Table: 200,000 simulated A/A tests.
A Bayesian probability honestly describes what you know on any evening. But the rule "stop at 95%" once again picks the peak of the walk. The Bayesian answer is different: decide by expected loss, that is, how many conversions you lose if you pick the wrong page. When the pages are almost the same, the loss from a mistake is small too. That is a sensible business rule, but it does not promise 5% false winners.
The honest answer: there is no free way. Every tool below costs something: more visitors, bolder changes, a weaker conclusion or risk. Example numbers: 5% conversion, we want to detect +10%, the usual standard (5% false winners, 80% power). How many visitors that takes, and why, is explained on the first page of the series.
Choose the number of looks and the days in advance, and use stricter bounds. O'Brien–Fleming lets you stop early if the effect is big and takes almost nothing away at the end. Choose the number of looks:
| Rule | False winners | Bound |z| at the last look | Max sample | Average sample if B really is +10% |
|---|
Price: planning for 3–30% more visitors at most, and discipline: looks only on the planned days. In return, when the effect is real, the test ends 18–22% earlier on average. Samples are relative to a single look with 80% power.
Price: you won't see small improvements. A bold change is harder to come up with, and if it only gives +3%, the test will miss it.
If the same users visited the site before the test, their "before" behaviour explains part of the spread "during". The CUPED method (Alex Deng, Ya Xu, Ron Kohavi, Toby Walker, Microsoft, 2013) subtracts this predictable part. The remaining spread is multiplied by 1 − ρ², where ρ is the correlation of the metric before and during the test.
Price: you need data on the same people before the test. New visitors from ads have none, and a binary "bought / didn't" usually correlates weakly with the past. At Bing the method cut the variance roughly in half.
Price: it is a proxy. More clicks on a button do not guarantee more money: a flashy button can bring in the curious, who don't buy. Check the link between the proxy and revenue separately.
| Standard | Sample | False | Missed |
|---|---|---|---|
| 95% and power 80% | 100% | 5% | 20% |
| 90% and power 80% | 79% | 10% | 20% |
| 95% and power 50% | 49% | 5% | 50% |
| 90% and power 50% | 34% | 10% | 50% |
"False" is the share of A/A tests with a winner, "missed" is the share of real +10% effects the test fails to see. For our example that is 31,234, 24,603, 15,288 and 10,767 visitors per page.
Price: this is a business decision, not a statistics trick. It is honest if you write the standard down before the start and look once. For cheap changes that are easy to roll back, 90% is often perfectly reasonable.
Thompson sampling splits the traffic each day in proportion to each page's chance of being the better one. Simulation: 2,000 visitors a day, 30 days, A gives 5%.
Price: a weaker conclusion. The worse page gets few visitors, so you know its conversion poorly. If the effect changes over time (novelty, weekends), the bandit can lock onto an accidental leader.
Sometimes there is too little traffic for any test. Then it is more honest to decide on judgement, user research and experience, ship the change and watch the metrics. Keep a holdout: a small share of traffic that keeps seeing the old version for another month.
| 10% holdout, 2,000 people a day, 30 days | Detected |
|---|---|
| New version 30% worse | > 99.9% |
| 20% worse | 96% |
| 10% better | 37% |
| 5% better | 13% |
Price: you won't learn whether the change helped. A holdout catches a disaster, not a small gain. But you don't fool yourself with a "95%" that isn't there.
Thirty looks at one test and twenty segments of one test work the same way: the more chances you give the noise, the more often it wins. The only difference is that evenings resemble each other while independent segments don't, so segments inflate the error even more. That is the third page of the series, "Multiple testing". And how noise creates "winners" even with a single look is shown on the first one, "Two identical pages".
Abraham Wald developed sequential analysis in 1943 in Columbia University's Statistical Research Group, when the US Navy asked for help with quality control of munitions. The method saved so many inspections that its use was restricted during the war. Wald published it openly in 1945 in the Annals of Mathematical Statistics.
Peter Armitage, C. McPherson and B. Rowe, in "Repeated significance tests on accumulating data" (Journal of the Royal Statistical Society, Series A, 1969), computed how repeated checks at the 5% level keep increasing the chance of a false alarm. Their calculations became the basis for the bounds later proposed by Pocock and by O'Brien and Fleming.
Stuart Pocock published group sequential methods in Biometrika in 1977, and Peter O'Brien and Thomas Fleming their scheme in Biometrics in 1979. Interim analyses of drug trials have used such bounds ever since: a committee can stop a trial early without inflating the chance of a false discovery.
That is the title of an essay by Evan Miller dated 18 April 2010, after which web developers and marketers started talking about the peeking problem. His main advice: fix the sample size in advance and don't stop the test early, or use a sequential or Bayesian design.
On 21 January 2015 Optimizely moved all customers to Stats Engine, built with statisticians from Stanford: sequential tests plus control of the false discovery rate. In its explanation the company gave an example: if you look at a test every 500 visitors, the chance of falsely declaring a winner rises to 26%.
Dirk Bassler and colleagues (JAMA, 2010) compared 91 clinical trials stopped early for benefit with 424 similar trials that ran to the end. The stopped ones showed a stronger effect: a ratio of relative risks of 0.71. The overestimate was largest in trials with fewer than 500 events, that is, where there was the least data.
Share your own explanation, an objection or a variant of the puzzle. You need a GitHub account to comment.
Is “Advanced” the right level for this puzzle?
Thank you! This helps set the levels right.
I keep adding new puzzles, new ways to explain them and write-ups of the discussions they start. Leave your email and I'll write when something new is out. No spam, one-click unsubscribe.
The sign-up form will appear here soon.