One person in 100 has the disease. The test catches 99% of sick people and wrongly flags only 1% of healthy ones. You test positive. Most people, many doctors included, conclude: "So I'm 99% likely to be ill." The real chance is 50/50. Check it on a crowd of 1,000, with a simulation and with Bayes' theorem.
This puzzle is known as the false positive paradox, and the mistake intuition makes here is called the base rate fallacy. We look at how accurate the test is and forget how rare the disease is. Bayes' theorem shows how to count it correctly.
1% of people have the disease, one person in 100. It has no visible symptoms.
If a person is ill, the test says "positive" 99% of the time. That's the sensitivity. If a person is healthy, it says "negative" 99% of the time. That's the specificity. So only 1% of healthy people get a false positive.
You're tested during a routine check-up, with no complaints. So you're a random member of the population.
The test is positive. What is the chance you're really ill?
"The test is 99% accurate, so I'm 99% likely to be ill"
In fact the chance is 50%. For every sick person who tests positive there is one healthy person with a false positive. Healthy people outnumber the sick 99 to 1, so even 1% of them adds up to as many positives as all the sick people together.
The numbers are illustrative. Real tests vary in accuracy, and doctors also weigh symptoms and other results. Round 5 shows how that changes the answer.
Lock in your answer before reading on. 1% of people have the disease, the test is 99% accurate, and your result is positive.
Each cell is a person. Change the share of sick people and the test's accuracy, then pick random people who tested positive and see who they really are.
| Test + | Test − |
|---|
The computer creates 100,000 people, each ill with the probability you set, and tests them all. The line is Bayes' theorem; the dots are the simulation. The horizontal axis is logarithmic, from 1 person in 10,000 to half the population.
The test answers "how likely is a positive result if a person is ill?". You want the reverse: "how likely am I to be ill if my result is positive?". These are different probabilities, and the base rate connects them.
Take 10,000 people. 100 of them are ill, and the test finds 99. 9,900 are healthy, and 1% of them, 99 people, get a false positive. That's 198 positives in total, 99 of them ill. Exactly half.
The left branch is thin, since only 1% are ill, but the test almost always fires on it. The right branch is thick, since 99% are healthy, and even a rare error there yields as many positives.
The test is always the same here: 99% sensitivity and specificity.
A positive result is 99 times more likely in an ill person than in a healthy one. So it multiplies the odds of disease by 99. But the starting odds were 1 to 99, and multiplying by 99 gives just 1 to 1. If 1 person in 10,000 has the disease, odds of 1 : 9,999 become 99 : 9,999, roughly 1 : 101.
S means sick, H means healthy. P(+ | S) is the test's sensitivity, P(+ | H) is 1 minus the specificity, and P(S) is the base rate. The denominator is the share of all positive results. When the disease is rare, the second term dominates it, and that term is the false positives. For a second test, the previous answer, 50%, becomes the new base rate.
The answer depends on the probability you start from. A second independent test, symptoms or a very rare disease change it many times over. The same maths applies to spam filters and to systems that flag fraudulent payments.
The scenario numbers are illustrative. They show the shape of the problem, not the accuracy of any real test.
The bars show the probability after 0 to 4 positive results in a row, assuming the tests err independently. Repeating the same kind of test often breaks that assumption, which is why confirmation usually uses a different method.
A test with 99% sensitivity and specificity. Press "Check": a simulation of 2,000,000 people counts the share who are ill among those with the given results.
| Situation | Chance of illness |
|---|
In 1978 Casscells, Schoenberger and Graboys put this question to 60 students and doctors at Harvard Medical School: a disease affects 1 person in 1,000, and the test has a 5% false positive rate. What is the chance that a person who tests positive is ill? The most common answer was 95%. Fewer than one in five gave the right answer, about 2%.
The psychologist Gerd Gigerenzer showed that people, doctors included, get the answer right far more often when the same problem is stated in counts of people ("99 of 100 ill people") instead of percentages and conditional probabilities. That's why round 2 shows a crowd, not a formula.
In 1999 in the UK, Sally Clark was convicted over the deaths of her two baby sons. A paediatrician told the court that the chance of two sudden infant deaths in one family was 1 in 73 million. The figure came from squaring the chance of one such death, as if the two deaths were independent. On top of that, the rarity of the event was confused with the probability of innocence. This is the prosecutor's fallacy, a close relative of the base rate fallacy. The conviction was overturned in 2003.
The English minister and mathematician Thomas Bayes died in 1761. His friend Richard Price found his work on inverse probability among his papers and presented it to the Royal Society. The essay was published in 1763. Pierre-Simon Laplace later gave the formula its familiar form.
In 2002 the programmer Paul Graham published the essay "A Plan for Spam", proposing to filter mail with Bayes' theorem: every word in a message shifts the chance that it's spam. The idea caught on quickly. A filter faces the same problem as a medical test: the result depends on what share of all mail is spam in the first place.
Daniel Kahneman and Amos Tversky gave people this problem. In a city, 85% of cabs are green and 15% are blue. A witness to a night-time accident says the cab was blue, and he identifies colours correctly 80% of the time. The most common answer is that the cab is blue with probability 80%. The truth is about 41%: green cabs are so much more common that the witness mistakes a green cab more often than he correctly sees a blue one.
Share your own explanation, an objection or a variant of the puzzle. Commenting needs a GitHub account.
I keep adding new puzzles, new ways to explain them and write-ups of the discussions they start. Leave your email and I'll write when something new is out. No spam, one-click unsubscribe.
The sign-up form will appear here soon.