Glossary / Standard error
You compute the average order value for the week and want to know how far that number can be trusted. It seems that twice as many orders would make it twice as precise. No: it only becomes twice as precise with four times as many orders. How much an average wobbles from one sample to the next is called the standard error.
“In plain words” is hidden on all puzzles.
Orders in a shop vary a lot: one person buys for $8, another for $150, and on average it comes to $40. The top chart shows 2,000 single orders. The bottom one shows 2,000 "weeks": each time we take n orders and compute their average order value. Increase n fourfold and watch the width of the lower pile.
Single orders
Average order of the week
An explanation without formulas, slightly simplified. The upper pile describes customers, and its width does not change: people don't become more alike because you counted more of them. The lower pile describes your estimate. In an average of many orders, big and small orders partly cancel each other, so the average wobbles less. But the cancelling is slow: the spread falls not n times but √n times. Four times the orders, a pile half as wide.
The typical spread of single values is the standard deviation σ. The typical spread of the average you computed from a sample is the standard error. It shows how far your number usually misses the true average.
An agency has 200 landing pages, and all are equally good: each one's true conversion rate is exactly 5%. Traffic differs: from 50 to 20,000 visitors a month. Each dot is one page; further right means more traffic (log scale). Who makes it into the "top 10 pages of the month" report?
top 10bottom 105% ± 2 standard errors
There is no difference between the pages at all, yet the report still finds "stars" and "laggards", and almost all of them are small. A small page has a large standard error, so its conversion flies far up and down. A big page stays close to the true 5%. The dashed funnel shows where ordinary fluctuations lie for each level of traffic.
“Rigorous” is hidden on all puzzles.
SE = σ / √n · for a conversion rate SE = √(p(1 − p) / n)
For the average order, σ is the standard deviation of single orders. For conversion, each visitor either bought (1) or not (0), and the σ of such zeros and ones is √(p(1 − p)). At 5% that is √0.0475 ≈ 0.218. The standard error of the conversion rate at 1,000 visitors: 0.218 / √1,000 ≈ 0.69 percentage points.
About 95% of samples give an estimate within ±2 standard errors of the truth (±1.96 to be exact). That is where the "margin of error ±…" in polls and in confidence intervals comes from. Here is what it means for a page with a true conversion rate of 5%:
| Visitors | Standard error | 95% of measurements between | Relative to 5% |
|---|
At 1,000 visitors, a page with a 5% conversion rate can easily show anything from 3.6% to 6.4%. One more thing: the error of the difference between two independent estimates is √(SE₁² + SE₂²). For two identical pages that is √2 ≈ 1.41 times larger than for one. So in an A/B test with 1,000 visitors per variant, the ordinary wobble of the difference reaches ±1.9 points, which is a "lift" of up to ±38% between identical pages.
“Deeper” is hidden on all puzzles.
In 2007 the statistician Howard Wainer wrote an essay for American Scientist called "The Most Dangerous Equation". That is what he called the formula σ / √n, which he credited to Abraham de Moivre and his book Miscellanea Analytica (1730). It is dangerous because it was ignored for centuries.
The pages in the second simulation are the same story in marketing. A "best campaigns of the month" report with no adjustment for size is mostly a ranking of small campaigns' luck.
The formula σ / √n only works when observations are independent. In web analytics they often aren't: the same people come back several times. Here there are 10,000 visits, and each person makes k visits. One person in ten is "warm" and buys on 41% of visits, the rest on 1%, so the overall conversion rate is still 5%. The computer repeats the month 2,000 times and measures how much the conversion rate really wobbles.
2,000 monthswhat the formula promises for 10,000 independent visits
One person's visits resemble each other, so they bring less new information than separate people. For equal groups the error grows by a factor of √(1 + (k − 1)ρ), where ρ is how alike one person's visits are (here ρ ≈ 0.30). At 10 visits per person, 10,000 visits weigh as much as about 2,600–2,700 independent ones. That is why A/B tests split and count by users, not by sessions.
The middle value: line everyone up in order, and half are below it, half above.
A millionaire walks into a bar: the average income shoots up, the median barely moves.
The sum of all values divided by how many there are: what everyone would get if it were shared equally.
Almost everyone has more legs than average: the average is a little under two.
The average result you can expect if you repeat the same thing very many times.
A die's expected value is 3.5, yet no face shows 3.5.
A number from −1 to +1 showing how closely two quantities move together.
In summer both ice cream sales and drownings go up. Ice cream isn't to blame: the heat drives both.
A measure of spread: how far values typically stray from the mean. The standard deviation is its square root, in the same units.
Two towns with the same average temperature: one stays near +15 all year, the other swings from −20 to +50.
A bell-shaped distribution: most values sit near the mean, and the further away, the rarer.
Every ball on a Galton board falls at random, yet thousands of balls always build the same bell.
The more repetitions, the closer the share of an outcome gets to its true probability.
The share of heads creeps towards 50%, while the gap between heads and tails in coins usually grows.
The chance of an event given that something else is already known.
Almost all NBA players are tall, but almost all tall people are not NBA players.
Events are independent if knowing about one doesn't change the chances of the other.
After five heads in a row tails is no more likely: the coin remembers nothing.
How many times you multiply the base by itself to get a number: log₂ 8 = 3, because 2 · 2 · 2 = 8.
A sheet of paper folded in half 42 times would be thicker than the distance to the Moon.
A sample some people get into more easily than others, so it describes everyone badly.
In 1936 a magazine polled millions of Americans and still picked the wrong election winner.
How often chance alone would produce a gap at least as big as yours if there were really no difference.
When there is no effect at all, p below 0.05 turns up as often as p between 0.50 and 0.55.
A range around a measured number, built by a method that covers the true value in a chosen share of samples, such as 95%.
Of twenty honest 95% intervals, on average one misses the truth, and none of them knows whether it is the one.
The probability that a test notices a real effect of a given size. A power of 80% means the test misses one working effect in five.
A weak A/B test that does find an effect almost always exaggerates it: a real +10% shows up on the dashboard as +24%.