Glossary / Sampling bias
In 1936 the American magazine Literary Digest mailed about 10 million ballots and got 2.4 million back. Its forecast: Alf Landon would take 57% and become president. In fact Franklin Roosevelt won 60.8% of the vote and carried 46 of the 48 states. George Gallup asked about 50,000 people, almost 50 times fewer, and named the right winner. It wasn't about how many answers there were, but whose answers they were.
“In plain words” is hidden on all puzzles.
A town has launched a new tram route. Four teams want to find out whether people like it. The first asks random passers-by. The second phones home numbers during the day. The third reads the reviews on the council's page. The fourth shares a form in a tram fans' club, and whoever wants fills it in. Collect answers and watch where the lines creep.
In fact 40% of the townspeople like the route. The answers axis is logarithmic: each mark is ten times further than the previous one.
“Rigorous” is hidden on all puzzles.
A sample is biased when the way it is chosen gives some people a better chance of getting in than others, and those chances are linked to what we are measuring. If everyone has the same chance, the sample is random, and on average it isn't wrong.
A small example. 40% of people like the route. Those who like it answer twice as eagerly. For every 100 townspeople we get 40 · 2 = 80 "votes" for and 60 against. The survey will show:
80 / (80 + 60) = 80 / 140 ≈ 57% instead of 40%
And it will be so for any number of answers: the doubled enthusiasm doesn't depend on how many people we asked.
The magazine took its addresses from its own subscriber lists, telephone books and car registrations. In the depths of the Great Depression, telephones and cars belonged mostly to better-off families, and they voted against Roosevelt more often. That is the first trap: who can be reached at all. The second turned out to be even bigger: only a quarter of those mailed sent the ballot back. A later analysis (Peverill Squire, 1988) showed that Roosevelt's opponents were keener to reply, and this did the most damage to the forecast.
The same town. We run 200 surveys of each kind: 100 random passers-by, or 10,000 answers to daytime calls, a hundred times more. Each survey is one tick on the chart.
“Deeper” is hidden on all puzzles.
p̂ − p = (E[p̂] − p) + (p̂ − E[p̂]), E[(p̂ − p)²] = bias² + q(1 − q) / n
Here p is the true share, p̂ the share in the sample, and q = E[p̂] the share among those who actually get into the sample. The first term is the bias q − p, the second is noise. Noise shrinks like 1/√n: a sample four times bigger scatters half as much. The bias does not depend on n at all. For the calls in our town q = 60% and p = 40%:
| Answers | Noise (standard deviation) | Bias |
|---|---|---|
| 100 | 4.9 points | 20 points |
| 10,000 | 0.49 points | 20 points |
| 1,000,000 | 0.049 points | 20 points |
A million answers give a very precise estimate of the wrong number. That is exactly what the 2.4 million ballots of 1936 were.
p̂w = Σg Ng/N · p̂g
In our town a quarter of the people are pensioners. Among them 76% like the route, among the rest 28%: together 0.25 · 76% + 0.75 · 28% = 40%. If the sample has too many pensioners, you can compute the share separately in each group g and add them up with the weights the groups have in the town. Pollsters do this with age, sex and region. Here is what weighting by age does to the answers collected at the top:
| Survey | Pensioners in the sample | As is | Weighted by age |
|---|
Weighting rescues the calls almost completely: there, only who was at home is distorted, and pensioners who picked up think the same as all pensioners. It barely fixes the reviews and the fans' form: there people are selected by their opinion of the route itself, and within every age group it is the unhappy ones or the fans who write. Weighting works only when we know which feature makes people drop out and know its true distribution. If dropping out depends on the very thing we measure, no formula knows any more how much is missing.
If the table shows dashes, collect at least a few dozen answers at the top.
The middle value: line everyone up in order, and half are below it, half above.
A millionaire walks into a bar: the average income shoots up, the median barely moves.
The sum of all values divided by how many there are: what everyone would get if it were shared equally.
Almost everyone has more legs than average: the average is a little under two.
The average result you can expect if you repeat the same thing very many times.
A die's expected value is 3.5, yet no face shows 3.5.
A number from −1 to +1 showing how closely two quantities move together.
In summer both ice cream sales and drownings go up. Ice cream isn't to blame: the heat drives both.
A measure of spread: how far values typically stray from the mean. The standard deviation is its square root, in the same units.
Two towns with the same average temperature: one stays near +15 all year, the other swings from −20 to +50.
A bell-shaped distribution: most values sit near the mean, and the further away, the rarer.
Every ball on a Galton board falls at random, yet thousands of balls always build the same bell.
The more repetitions, the closer the share of an outcome gets to its true probability.
The share of heads creeps towards 50%, while the gap between heads and tails in coins usually grows.
The chance of an event given that something else is already known.
Almost all NBA players are tall, but almost all tall people are not NBA players.
Events are independent if knowing about one doesn't change the chances of the other.
After five heads in a row tails is no more likely: the coin remembers nothing.
How many times you multiply the base by itself to get a number: log₂ 8 = 3, because 2 · 2 · 2 = 8.
A sheet of paper folded in half 42 times would be thicker than the distance to the Moon.
How often chance alone would produce a gap at least as big as yours if there were really no difference.
When there is no effect at all, p below 0.05 turns up as often as p between 0.50 and 0.55.
A range around a measured number, built by a method that covers the true value in a chosen share of samples, such as 95%.
Of twenty honest 95% intervals, on average one misses the truth, and none of them knows whether it is the one.
The probability that a test notices a real effect of a given size. A power of 80% means the test misses one working effect in five.
A weak A/B test that does find an effect almost always exaggerates it: a real +10% shows up on the dashboard as +24%.
How far an average or a rate computed from a sample usually misses the truth: σ divided by the square root of the sample size.
Halving the error takes four times as much data.