Glossary / p-value
You show the same page to two halves of your visitors and compute p, a number from 0 to 1 that every A/B testing tool reports. It seems that with no difference, p should be close to 1. In fact every value is equally likely: p below 0.05 turns up as often as p between 0.50 and 0.55. That is why even between identical pages about one test in twenty looks "significant".
“In plain words” is hidden on all puzzles.
Each test here is two versions of a landing page with 10,000 visitors each and a 5% base conversion rate. The computer computes p the way ordinary tools do (a two-proportion z-test) and drops it into one of 20 bars. Start with identical pages, then give version B a real advantage.
p < 0.05: the tool says "significant"the other testsexact calculation
This is a simplified explanation, but it holds the whole idea. p answers one question: if the pages were identical, how often would chance alone produce a gap at least as big as the one you see? p = 0.03 means "three times in a hundred". The smaller p is, the stranger your result would be in a world with no difference. The 0.05 line, "one time in twenty", is just a convention.
That is where the flat histogram comes from. In a world with no difference your test is just one more random draw from the very draws it is compared against. So it is equally likely to land among the 5% strangest, the 5% most ordinary, or anywhere in between.
“Rigorous” is hidden on all puzzles.
Version A: 500 conversions out of 10,000 visitors, 5.0%. Move how many conversions B collected. The bell shows how the difference B − A would scatter if the pages were identical. The shaded area of the tails, where the gap is at least as large as yours (in either direction), is p.
p̄ = (kA + kB) / 2n · SE = √(p̄ (1 − p̄) · 2 / n) · z = (kB − kA) / n / SE · p = 2 · (1 − Φ(|z|))
SE is the standard error of the difference: how far B − A usually wanders by chance alone. Φ is the area under the normal curve to the left of z. At |z| = 1.96 you get exactly p = 0.05.
The 0.05 line promises something else: of the ideas that really do nothing, 5% will slip through the filter. What share of all "winners" is false depends on how many ideas really work and how often the test spots a real effect (its power). Here are 1,000 ideas a team checked with A/B tests.
works, the test spotted itworks, the test missed itdoesn't work, yet p < 0.05doesn't work, the test stays quiet
This is the same logic as in the false positive puzzle: an accurate filter applied to a rare event gives many false alarms. The rarer real improvements are, the more "winners" are noise.
“Deeper” is hidden on all puzzles.
p is the share of possible results of the "no difference world" that are stranger than yours. So p ≤ 0.2 exactly when your result landed among the strangest 20%. If there really is no difference, your result is drawn from that same distribution, so it lands in those 20% with probability exactly 0.2. The same for any threshold t: P(p ≤ t) = t. That is the uniform distribution from 0 to 1. In the same way, your height as a percentile of a random crowd is equally likely to be any percentile.
The exact calculation for identical pages gives 4.99% of tests with p < 0.05 and 4.8–5.3% in each of the following bars. The last one, from 0.95 to 1, gets only 3.9%: conversions are whole numbers, and p above 0.95 only happens when the counts of A and B are equal or differ by one. Flatness is a property of the continuous model; on whole numbers it is almost exact.
When there is an effect, the distribution of p squeezes toward zero: with 10,000 visitors and a real +10%, p < 0.05 comes out in 35.4% of tests, with +20% in 87.4%. That share is called the power of the test.
The first known calculation of this kind was made by the Scottish physician John Arbuthnot in 1710. He went through London christening records for 82 years and saw that every year more boys than girls were christened. If the chances were equal, this would happen with probability (½)82, about 2 · 10−25. Arbuthnot concluded it was not chance.
The 0.05 line comes from Ronald Fisher, in his 1925 book "Statistical Methods for Research Workers". Fisher noted that p = 0.05 corresponds to a deviation of about 1.96, nearly two standard deviations, and that it is "convenient to take this point as a limit" in judging whether a deviation is significant. A convenient limit later hardened into a rule.
In March 2016 the American Statistical Association, for the first time on behalf of the whole community of statisticians, issued a statement on p-values (Ronald Wasserstein and Nicole Lazar, The American Statistician). Its six principles in our own words:
The middle value: line everyone up in order, and half are below it, half above.
A millionaire walks into a bar: the average income shoots up, the median barely moves.
The sum of all values divided by how many there are: what everyone would get if it were shared equally.
Almost everyone has more legs than average: the average is a little under two.
The average result you can expect if you repeat the same thing very many times.
A die's expected value is 3.5, yet no face shows 3.5.
A number from −1 to +1 showing how closely two quantities move together.
In summer both ice cream sales and drownings go up. Ice cream isn't to blame: the heat drives both.
A measure of spread: how far values typically stray from the mean. The standard deviation is its square root, in the same units.
Two towns with the same average temperature: one stays near +15 all year, the other swings from −20 to +50.
A bell-shaped distribution: most values sit near the mean, and the further away, the rarer.
Every ball on a Galton board falls at random, yet thousands of balls always build the same bell.
The more repetitions, the closer the share of an outcome gets to its true probability.
The share of heads creeps towards 50%, while the gap between heads and tails in coins usually grows.
The chance of an event given that something else is already known.
Almost all NBA players are tall, but almost all tall people are not NBA players.
Events are independent if knowing about one doesn't change the chances of the other.
After five heads in a row tails is no more likely: the coin remembers nothing.
How many times you multiply the base by itself to get a number: log₂ 8 = 3, because 2 · 2 · 2 = 8.
A sheet of paper folded in half 42 times would be thicker than the distance to the Moon.
A sample some people get into more easily than others, so it describes everyone badly.
In 1936 a magazine polled millions of Americans and still picked the wrong election winner.
A range around a measured number, built by a method that covers the true value in a chosen share of samples, such as 95%.
Of twenty honest 95% intervals, on average one misses the truth, and none of them knows whether it is the one.
The probability that a test notices a real effect of a given size. A power of 80% means the test misses one working effect in five.
A weak A/B test that does find an effect almost always exaggerates it: a real +10% shows up on the dashboard as +24%.
How far an average or a rate computed from a sample usually misses the truth: σ divided by the square root of the sample size.
Halving the error takes four times as much data.