Glossary / Confidence interval
A report says: "conversion 5.2%, 95% confidence interval 3.8% to 6.6%". The interval is an honest admission: we don't know the exact number, here is a range. But the 95% is not about this range, it is about the method. If you repeat the measurement again and again, about 19 intervals in 20 will cover the true conversion rate and one will miss. And from the inside, none of them knows whether it is that twentieth one.
“In plain words” is hidden on all puzzles.
The page's true conversion rate is 5.0%. Only the simulation knows it. A hundred marketers each measure it on their own sample of visitors and each draws an interval. The vertical line is the truth. The intervals that missed it are coloured.
covers the truthmissesthe true 5.0%
This is a simplified picture, but an accurate one. The method that builds the interval is a net cast into the water. The true number is a fish that stays put. The 95% is the quality of the net: out of every 100 casts, 95 catch the fish. Once the net is pulled out, the fish is either in it or not; "95% chance" no longer applies to this particular catch. But without knowing where the fish is, you can't see whether this catch was a good one. Press "Hide the truth" above: the intervals that missed look exactly like the rest.
“Rigorous” is hidden on all puzzles.
p̂ ± z · √(p̂ (1 − p̂) / n) · z = 1.28 (80%), 1.64 (90%), 1.96 (95%), 2.58 (99%)
p̂ is the measured conversion rate, n the number of visitors. The square root is the standard error SE: how far the measured rate usually strays from the true one. z says how many such errors to lay off on each side so that the normal curve covers the chosen share. Here is half the width of a 95% interval at a 5% conversion rate:
| Visitors | Conversions | ± in pp | ± of 5% |
|---|---|---|---|
| 1,000 | 50 | 1.35 | 27% |
| 4,000 | 200 | 0.68 | 13.5% |
| 16,000 | 800 | 0.34 | 6.8% |
With 1,000 visitors, "5%" really means "somewhere from 3.6% to 6.4%". The "±3%" printed under poll results is also half the width of a 95% interval. How many people a poll needs for that, and why it barely depends on the size of the country, is the subject of the election poll puzzle.
A common rule: "if the intervals of A and B overlap, there is no difference". It is wrong. Version A: 500 conversions out of 10,000. Move B's result. At the top are each version's intervals, at the bottom the interval of the difference B − A. Significance is decided by the bottom one: whether it covers zero.
Why: the uncertainties of A and B add not as segments but as squares. The interval of the difference is only √2 ≈ 1.41 times wider than each one, not twice. So the intervals can overlap a little while the difference is already significant: here that is B from 563 to 588 conversions. If the intervals don't overlap at all, the difference is certainly significant. Nathaniel Schenker and Jane Gentleman pointed out this trap in a 2001 article in The American Statistician.
“Deeper” is hidden on all puzzles.
The formula p̂ ± 1.96 · SE leans on the normal curve, while conversions are whole numbers. The exact calculation for 5% shows that the "95%" interval with 1,000 visitors covers the truth in 94.2% of samples, and with 100 visitors (only five expected conversions) in 87.7%. On top of that, coverage jumps like a saw: add a dozen visitors and it can shift by several points. Lawrence Brown, Tony Cai and Anirban DasGupta showed in Statistical Science in 2001 that this chaos is more persistent than textbooks claimed, and recommended the Wilson interval.
simple interval p̂ ± 1.96 · SEWilson intervalthe promised 95%
True conversion 5%, from 30 to 500 visitors. Each point is the exact sum of binomial probabilities of all outcomes whose intervals cover 5%.
Wilson: (p̂ + z²/2n ± z · √(p̂ (1 − p̂)/n + z²/4n²)) / (1 + z²/n)
The theory of confidence intervals was created by Jerzy Neyman. He was born in 1894 in Bendery and from 1912 studied at Kharkiv University, where Serhiy Bernstein taught him probability. Neyman published the main paper, "Outline of a Theory of Statistical Estimation Based on the Classical Theory of Probability", in 1937 in the Philosophical Transactions of the Royal Society. That is where the idea shown by the simulation above appeared: the probability is not about the unknown number but about the procedure that builds intervals again and again.
You can say "a 95% chance the truth is in this interval" in a different, Bayesian approach: there the unknown number has a distribution even before measuring, and a different, Bayesian kind of interval is computed with it. With plenty of data the two intervals usually almost coincide, so the everyday reading rarely does harm. The mistakes happen where data are scarce.
The middle value: line everyone up in order, and half are below it, half above.
A millionaire walks into a bar: the average income shoots up, the median barely moves.
The sum of all values divided by how many there are: what everyone would get if it were shared equally.
Almost everyone has more legs than average: the average is a little under two.
The average result you can expect if you repeat the same thing very many times.
A die's expected value is 3.5, yet no face shows 3.5.
A number from −1 to +1 showing how closely two quantities move together.
In summer both ice cream sales and drownings go up. Ice cream isn't to blame: the heat drives both.
A measure of spread: how far values typically stray from the mean. The standard deviation is its square root, in the same units.
Two towns with the same average temperature: one stays near +15 all year, the other swings from −20 to +50.
A bell-shaped distribution: most values sit near the mean, and the further away, the rarer.
Every ball on a Galton board falls at random, yet thousands of balls always build the same bell.
The more repetitions, the closer the share of an outcome gets to its true probability.
The share of heads creeps towards 50%, while the gap between heads and tails in coins usually grows.
The chance of an event given that something else is already known.
Almost all NBA players are tall, but almost all tall people are not NBA players.
Events are independent if knowing about one doesn't change the chances of the other.
After five heads in a row tails is no more likely: the coin remembers nothing.
How many times you multiply the base by itself to get a number: log₂ 8 = 3, because 2 · 2 · 2 = 8.
A sheet of paper folded in half 42 times would be thicker than the distance to the Moon.
A sample some people get into more easily than others, so it describes everyone badly.
In 1936 a magazine polled millions of Americans and still picked the wrong election winner.
How often chance alone would produce a gap at least as big as yours if there were really no difference.
When there is no effect at all, p below 0.05 turns up as often as p between 0.50 and 0.55.
The probability that a test notices a real effect of a given size. A power of 80% means the test misses one working effect in five.
A weak A/B test that does find an effect almost always exaggerates it: a real +10% shows up on the dashboard as +24%.
How far an average or a rate computed from a sample usually misses the truth: σ divided by the square root of the sample size.
Halving the error takes four times as much data.