Glossary / Bayes' theorem
At night a cab hits a pedestrian and drives off. The town has two companies: 85% of the cabs are green, 15% are blue. A witness says the car was blue. He was tested in the same conditions and names the colour correctly 80% of the time. Most people answer at once: "So it is 80% blue." This is a classic problem by Daniel Kahneman and Amos Tversky, and intuition misses here by almost a factor of two. Bayes' theorem is the rule for honestly changing your mind when new evidence arrives.
“In plain words” is hidden on all puzzles.
Here are all 100 cabs in town. Imagine any of them could have caused the accident, and the witness looked at each one. Press the buttons in turn and count the cars he would call blue.
The witness doesn't lie and doesn't err more often than stated. There are simply almost six times as many green cars, and even rare mistakes on so many of them produce more "blues" than the real blue cabs do. The witness's words raised the chance of a blue cab from 15% to 41%, but not to 80%.
“Rigorous” is hidden on all puzzles.
Bayes' theorem links three quantities. The hypothesis H is "the cab was blue", the evidence E is "the witness said 'blue'".
What we thought before the evidence. P(blue) = 0.15.
How often this evidence turns up under each hypothesis. P(says "blue" | blue) = 0.8, P(says "blue" | green) = 0.2.
What we think after the evidence. P(blue | says "blue") is what we are after.
P(H | E) = P(E | H) · P(H) / P(E)
P(blue | "blue") = 0.8 · 0.15 / (0.8 · 0.15 + 0.2 · 0.85) = 0.12 / 0.29 ≈ 0.41
The denominator is every case in which the witness says "blue": 12 real blue cabs and 17 green ones he got wrong, 29 out of 100 in all. Where the formula itself comes from is explained in the article on conditional probability. Here we care about something else: how to use it to change your mind.
The police found another witness, just as reliable, who saw everything from across the street and talked to no one. He says "blue" too. Now 41% is the starting belief, and the new evidence shifts that. Simulate 10,000 night accidents: in each, a random cab and several independent witnesses. We keep only the nights on which the witnesses said what you chose.
Two weak pieces of evidence turned out strong together: neither alone reaches even half, yet together they give almost three quarters. And two witnesses who contradict each other bring us back to exactly 15%: their voices cancelled out.
"Extraordinary claims require extraordinary evidence": Carl Sagan popularised the phrase, and Pierre-Simon Laplace put the idea forward much earlier. Bayes explains why. The same witness is equally convincing whether he says "I saw a fox in the park" or "I saw a tiger in the park". But the starting odds of these events differ, and that decides everything. Choose how rare the event is and see how many independent witnesses, each 80% reliable, it takes.
One striking piece of evidence weighs little if the event is almost impossible: from a one-in-a-million chance it only gets you to four in a million. To pass one half you need about ten independent witnesses, because each only multiplies the odds by four, and they have to be multiplied a millionfold.
“Deeper” is hidden on all puzzles.
P(H | E) / P(not H | E) = P(H) / P(not H) · P(E | H) / P(E | not H)
Write Bayes' formula for H and for "not H" and divide one by the other: the awkward denominator P(E) cancels. A single multiplication remains. For the cab, the starting odds of blue are 15 : 85, and the likelihood ratio (LR) is 0.8 / 0.2 = 4. After one witness the odds are 60 : 85 = 12 : 17, that is 12 / 29 ≈ 41%. Each further independent witness who says "blue" multiplies the odds by 4 again, and one who says "green" divides them by 4.
For the choice in the simulation above: –
log2 posterior odds = log2 prior odds + log2 LR1 + log2 LR2 + …
A logarithm turns multiplication into addition. So if you measure confidence by the logarithm of the odds, each independent piece of evidence simply adds its own number. One witness gives log2 4 = 2 bits in favour of a blue cab. The starting belief is log2(15/85) ≈ −2.5 bits, that is 2.5 bits against. Zero means "fifty-fifty". Turing used the base-10 logarithm and counted in decibans, tenths of a ban: a witness gives 10 · log10 4 ≈ 6 decibans. His colleague I. J. Good wrote that a change of about 1 deciban is about the smallest a person can feel.
| "Blue" witnesses | Bits | Odds | Probability |
|---|
Each witness moves the point by the same step of 2 bits. So the line is even, though the probability grows unevenly: 15%, 41%, 74%, 92%. Near 50% one step changes a lot, near 100% almost nothing.
LR = P(E | H) / P(E | not H)
The strength of evidence is not how likely it is under your version, but the ratio of two numbers. "The cab had a taxi sign on the roof" is 100% true for a blue cab. But it is also 100% true for a green one, so LR = 1, the logarithm is 0 bits, and the belief doesn't move a single step. Conversely, evidence that is rare under both versions but ten times rarer under one of them weighs 10 · log10 10 = 10 decibans, however mundane it sounds.
P(E1, E2 | H) = P(E1 | H) · P(E2 | H)
Likelihood ratios may be multiplied only when the pieces of evidence are independent under each hypothesis. If the second witness first heard the first one, his "blue" is already half a retelling, and the real LR of the second testimony is less than 4. In the extreme case, when he simply repeats what he heard, the LR is 1, and two voices weigh as one. The order of independent evidence, on the other hand, doesn't matter: multiplication is commutative, so it makes no difference which witness you hear first.
The theorem is named after Thomas Bayes, an English minister and mathematician. His essay was published by Richard Price in 1763, after the author's death.
The middle value: line everyone up in order, and half are below it, half above.
A millionaire walks into a bar: the average income shoots up, the median barely moves.
The sum of all values divided by how many there are: what everyone would get if it were shared equally.
Almost everyone has more legs than average: the average is a little under two.
The average result you can expect if you repeat the same thing very many times.
A die's expected value is 3.5, yet no face shows 3.5.
A number from −1 to +1 showing how closely two quantities move together.
In summer both ice cream sales and drownings go up. Ice cream isn't to blame: the heat drives both.
A measure of spread: how far values typically stray from the mean. The standard deviation is its square root, in the same units.
Two towns with the same average temperature: one stays near +15 all year, the other swings from −20 to +50.
A bell-shaped distribution: most values sit near the mean, and the further away, the rarer.
Every ball on a Galton board falls at random, yet thousands of balls always build the same bell.
The more repetitions, the closer the share of an outcome gets to its true probability.
The share of heads creeps towards 50%, while the gap between heads and tails in coins usually grows.
The chance of an event given that something else is already known.
Almost all NBA players are tall, but almost all tall people are not NBA players.
Events are independent if knowing about one doesn't change the chances of the other.
After five heads in a row tails is no more likely: the coin remembers nothing.
How many times you multiply the base by itself to get a number: log₂ 8 = 3, because 2 · 2 · 2 = 8.
A sheet of paper folded in half 42 times would be thicker than the distance to the Moon.
A sample some people get into more easily than others, so it describes everyone badly.
In 1936 a magazine polled millions of Americans and still picked the wrong election winner.
How often chance alone would produce a gap at least as big as yours if there were really no difference.
When there is no effect at all, p below 0.05 turns up as often as p between 0.50 and 0.55.
A range around a measured number, built by a method that covers the true value in a chosen share of samples, such as 95%.
Of twenty honest 95% intervals, on average one misses the truth, and none of them knows whether it is the one.
The probability that a test notices a real effect of a given size. A power of 80% means the test misses one working effect in five.
A weak A/B test that does find an effect almost always exaggerates it: a real +10% shows up on the dashboard as +24%.
How far an average or a rate computed from a sample usually misses the truth: σ divided by the square root of the sample size.
Halving the error takes four times as much data.
Counting how many ways things can be ordered or chosen: permutations n!, combinations C(n, k).
A well-shuffled deck is almost certainly in an order nobody has ever seen.
The sum 1 + ½ + ⅓ + ¼ + …: it grows without limit, but incredibly slowly.
Identical books can be stacked so the top one overhangs the table edge as far as you like.
Growth where a quantity is multiplied by the same factor each time, so it doubles at equal intervals.
One grain on the first square of a chessboard, doubling on each next one: the last square holds as much rice as the world grows in over 300 years.
A path where each next step is taken at random: forward or back, left or right.
A drunk man will always find his way home, but a drunk bird may never return.
A stream of independent random events at a constant average rate: calls, Geiger clicks, raindrops.
In 1944 Londoners thought German flying bombs were aimed at certain districts; it was ordinary random clumping.