A map of an imaginary country: 3,000 counties, from villages of a thousand people to cities of a million. The dark squares are the counties where a rare disease is rarest. Almost all of them are small and rural. Clean air? Or something else? Lock in your answer, then check it with a game, a simulation and the explanations.
The smallest counties have both the lowest and the highest rates, although the true risk is the same everywhere. Few people means few cases, and one case more or less throws the rate to an extreme. Abraham de Moivre captured this in an equation back in 1730. Check it with a game, a simulation and the explanations.
The statisticians Andrew Gelman and Deborah Nolan showed students a map of kidney-cancer rates in US counties. The counties with the lowest rates were mostly rural, in the Midwest, the South and the West. Here is the same story in an imaginary country where we know the whole truth.
3,000 counties. Populations run from 1,000 to 1,000,000: a third have up to 10,000 people, a third 10,000 to 100,000, a third 100,000 to a million.
A rare disease. Every case over ten years goes into the county's statistics.
Rate = cases ÷ population × 100,000. That is how counties of different sizes are compared: cases per 100,000 people.
The 10% of counties with the lowest rates are almost all small and rural. Why?
"The countryside has clean air, fresh food and less stress. That is why people get sick less"
But the 10% of counties with the highest rates are almost all small and rural too. What they share is not a way of life but size. A village of 1,000 expects 0.25 cases. Zero cases gives a rate of 0, and a single case gives 100 at once, four times the true risk.
Here are four explanations people usually offer. Which one best explains the map?
Pick an answer. The 10 healthiest and the 10 sickest counties of the imaginary country will appear here.
Hard to decide? Open the hints one at a time: the easy one first, then stronger ones.
Look not only at the healthiest counties but at the sickest too. If country air were the cause, where should villages land in the list of the sickest?
How many cases of a rare disease does a village of 1,000 expect if the risk is 25 per 100,000? And what rate per 100,000 does it show if one person falls ill?
Flip a coin 4 times, then 100 times. In which case is a "strange" share of heads easier to get, though the coin is the same?
Pick a county size and draw ten of them. Each has a true risk of 25 per 100,000, shown by the dashed line. See how far from it the dots of villages and cities land.
| People | Counties | Min | Max |
|---|
A county of 1,000 people can only show 0, 100, 200… per 100,000: it can never show the true 25. 78% of such counties have no case at all, and 22% show 100 or more.
Each dot is a county: population across, rate up. The funnel lines are computed exactly: between them lie 95% of the rates that chance alone produces for a county of that size. For small counties the funnel is wide, for big ones narrow.
The first layer has no formulas, the second has a table and the square-root rule, the third is history: from a twelfth-century mint to billion-dollar grants for small schools.
“In plain words” is hidden on all puzzles.
The rate is per 100,000 people. In a village of a thousand, one patient makes it 100 at once; in a city of a million, only 0.1. The village expects a quarter of a case, so it shows either 0 (the healthiest place in the country) or 100 and more (one of the sickest). There is no middle for it.
This is not the whole story: mid-sized counties are noisy too, just less so. The next cards show how the noise depends on size.
If country air were the cause, villages would gather only at the bottom of the ranking. Instead they fill both ends. When the same kind of place turns up among both the best and the worst, the first suspect should be the sample size, not a cause.
The coin is the same fair coin. But four flips easily produce a "biased coin", and a hundred almost never do. A small county is like a few flips, a big city like thousands.
About 450 of the 3,000 counties have no case at all. These are not 450 oases of health but 450 small counties that were lucky this decade. Next decade others will be lucky, and there will be just as many "oases", only elsewhere.
So you cannot compare a village with a city on the raw rate. Compare it with the spread that chance gives a county of the same size. That is exactly what the funnel in round 3 does.
“Rigorous” is hidden on all puzzles.
| People | Expected cases | +1 case | 95% of counties |
|---|---|---|---|
| 1k | 0.25 | 100 | 0–200 |
| 10k | 2.5 | 10 | 0–60 |
| 100k | 25 | 1 | 16–35 |
| 1M | 250 | 0.1 | 21.9–28.1 |
The true risk in every row is 25. The last column is computed exactly for a random number of cases: the limits that only 2.5% of counties fall below and 2.5% rise above. For a village the range runs from zero to eight times the true risk; for a city it is ±12%.
| Expected cases | √ | Usual swing |
|---|---|---|
| 25 | 5 | ±20% |
| 100 | 10 | ±10% |
| 400 | 20 | ±5% |
| 2,500 | 50 | ±2% |
The number of cases swings by about the square root of the expected number: expect 25 and you mostly get 20 to 30. So the relative swing is 1/√(expected cases). Halving it takes four times as many people; cutting it tenfold takes a hundred times as many. 95% of counties stay within about two such swings.
In a county of n people with risk p, the number of cases X follows a Poisson distribution almost exactly: its variance equals its mean.
The standard error of the rate X/n. This is de Moivre's equation σ/√n: the spread of an average falls as the square root of the sample size.
Approximate funnel limits and an example for 100,000 people (rate per 100,000, SE = 5). The exact Poisson limits give 16–35.
How many people it takes to narrow the limits k times.
The funnel in round 3 uses exact Poisson quantiles, not the approximation. Because cases come only in whole numbers, on average 2.9% of counties fall outside, not exactly 5%.
“Deeper” is hidden on all puzzles.
In 1730, in "Miscellanea Analytica", his work on the binomial distribution, Abraham de Moivre showed that the spread of an average shrinks not in proportion to the sample size but as its square root. In 2007 the statistician Howard Wainer, in "The Most Dangerous Equation" in American Scientist, named the formula σ/√n the most dangerous equation: not because knowing it does harm, but because not knowing it has cost billions.
Gelman and Nolan used the map of kidney-cancer rates in US counties in their book "Teaching Statistics: A Bag of Tricks" (2002). Wainer took it apart in his article: the counties in the lowest 10% and in the highest 10% lie side by side, in the same rural regions. Daniel Kahneman retold the example in chapter 10 of "Thinking, Fast and Slow" (2011), titled simply "The Law of Small Numbers".
Small schools are overrepresented among the best schools by test scores. In the late 1990s the Bill and Melinda Gates Foundation began backing small schools on a large scale, and other foundations joined in. Wainer and Harris Zwerling checked Pennsylvania's schools and found small schools overrepresented among the worst as well: the same spread as in the kidney-cancer counties. In his 2009 annual letter Bill Gates acknowledged that many of the small schools the foundation invested in had not improved students' achievement in any significant way.
What precision costs, how to build a ranking that does not put random villages on top, and where else small numbers hide.
De Moivre's equation holds for any average of independent measurements: a share of patients, a school's mean score, a site's conversion rate, a poll result. The spread of one measurement, σ, is divided by the square root of their number.
Hence the price of precision: each doubling of precision costs four times the data. On the ladder to the left the funnel width halves almost exactly at each step once more than a dozen cases are expected. On the lowest rungs it jumps unevenly, because cases come only in whole numbers. The same rule explains why a poll of 1,000 people has a margin of about ±3 points, and getting ±1.5 means asking 4,000 (the poll puzzle).
Now the counties' true risks differ a little: mostly within ±20% of 25. Build two "worst ten" lists: by the raw rate, and by an adjusted rate that adds 100,000 "average" people with 25 cases to every county.
| Raw | Adjusted |
|---|
Any ranking by a share or an average has the same trap: both its ends fill up with whoever has the least data. The best surgeon after 3 operations, the best product with two five-star reviews, the worst school with 20 pupils.
There are two cures. The funnel compares each county with the limits for its size and flags only those that leave them. The adjusted rate pulls small counties towards the overall average: their own data counts for little, while big cities barely move. This is the empirical Bayes approach. The price is that neither method will see a small county's real trouble while its data is thin.
The smallest lift a test can detect 8 times out of 10 at a 5% significance level. Change the base conversion rate and the visitors per variant.
| Visitors per variant, 3% conversion | Detectable lift |
|---|---|
| 1,000 | 71% |
| 10,000 | 22.5% |
| 100,000 | 7.1% |
Formula: 2.8 · √(2p(1 − p)/n) / p, where 2.8 ≈ 1.96 + 0.84. A site with 10,000 visitors per variant will almost never detect a +5% lift: the noise is bigger. So its published wins are big, and that same noise has often blown them up. More in the puzzles on the A/A test and multiple testing.
Since the twelfth century the English mint has undergone the Trial of the Pyx: a batch of coins is weighed against a standard. The tolerance for one coin was multiplied by the number of coins in the batch, not by its square root. The tolerance came out far too wide: as Wainer writes, for almost 600 years the mint could pick out the overweight coins for itself and the batch would still pass.
Tversky and Kahneman asked students: a large hospital has 45 births a day, a small one 15. Which will record more days a year with over 60% boys? 56% said "about the same". In fact such days are 15% of the year at the small hospital (about 55 days) and 7% at the large one (about 25).
Amos Tversky and Daniel Kahneman titled their 1971 paper in Psychological Bulletin "Belief in the law of small numbers", a twist on the law of large numbers. They showed that even research psychologists trained in statistics trust results from small samples too much.
That is how many schools with 5th grades Wainer and Zwerling checked. Of the 50 best on the reading test, 12% were among the smallest 3% of schools, four times what chance alone would give. But of the 50 worst, 18% were among the smallest too.
Wainer noticed that the ten safest US cities, the ten most dangerous and the ten largest have not a single city in common. Extreme crime rates, like extreme disease rates, come mostly from smaller places.
That much, according to Wainer, the Gates Foundation had given to education projects by 2001, at the height of its push for small schools. In 2005 the foundation began moving away from that bet, and in 2009 Bill Gates admitted in his annual letter that many of the small schools it had invested in had not raised students' achievement in any significant way.
Share your own explanation, an objection or a variant of the puzzle. You need a GitHub account to comment.
Is “School maths” the right level for this puzzle?
Thank you! This helps set the levels right.
I keep adding new puzzles, new ways to explain them and write-ups of the discussions they start. Leave your email and I'll write when something new is out. No spam, one-click unsubscribe.
The sign-up form will appear here soon.