A city has 500,000 adults. At the scene of a burglary the thief left blood, and its features occur in one adult in 10,000. During a check that had nothing to do with the case, a man gave a sample, and it matched. Nothing else points to him. The expert tells the court: "A random innocent person would match with a chance of only 1 in 10,000." The prosecutor concludes: "So the chance he is innocent is 1 in 10,000." Is the prosecutor right?
Among 500,000 adults about 50 other, innocent people share the same blood features. All of them would match just as well. So the match narrows the field to about 51 people, and only one of them is the thief: the chance of guilt is about 2%, and of innocence about 98%. The prosecutor swapped two different questions: "how likely is a match if the person is innocent" and "how likely is the person innocent if they match".
The case is made up, but the numbers are realistic and the mistake in it is real: innocent people have been convicted because of it. Lawyers call it "the prosecutor's fallacy". Behind it are conditional probability and Bayes' theorem.
A city has 500,000 adults. One of them broke into a shop at night, took the till and cut a hand on the shop window. Blood was left on the glass. There are no witnesses and no cameras.
The lab determined the blood type and a few other features. This combination occurs in one adult in 10,000. The lab makes no mistakes: the thief's blood is sure to match the thief's sample.
A week later a man gave a blood sample for a reason that had nothing to do with the burglary, and the features matched. He has no alibi, but nothing else points to him either: no stolen goods, no witnesses, no motive.
The expert: "If the defendant had nothing to do with it, such a match would happen with a chance of only 1 in 10,000." The prosecutor: "So the chance he is innocent is 1 in 10,000, that is 0.01%." What is the real probability that the defendant is innocent?
"A 1 in 10,000 match is nearly proof. An innocent man would not match like that"
A single innocent person really almost never matches. But the city has 499,999 innocent adults, and among them about 50 such "almost nevers" pile up. There is one thief. So among everyone who matches, the thief is just one of about 51.
A 1 in 10,000 match, 500,000 adults in the city, no other evidence. Pick an answer.
A breakdown of your answer will appear here.
Hard to decide? Open the hints one at a time: the easy one first, then stronger ones.
The expert gave the chance of a match for one innocent person. But there is more than one innocent person in the city.
Work out how many of the 499,999 innocent adults would share the blood features if each matches with a chance of 1 in 10,000.
Imagine the whole city was tested. The thief and all those innocent people would match. The defendant is one of them, and nothing else is known about him. What share of them is the thief?
Each case is a different place with different traces. The court's rule in this game: "guilty" only if the chance of guilt is at least 95%. Decide whether it is. You can ask the detective to count the people the trace would match, but it costs 4 points. A correct verdict earns 10 points. At the end of each case the game reveals who the thief in that town really was: sometimes even a correct verdict does not save you from a mistake.
It gives the solution away, so make your own guess first.
The simulation builds town after town. Each has one thief, and every other resident matches the trace by chance, with the same probability. Then it counts everyone who matched and checks what share of them are guilty. That is the chance of guilt for a person about whom we only know that they matched. Bayes' theorem gives this number exactly. Pick the town size and how rare the trace is.
It shows the answer right away.
The expert answered one question, while the court needs the answer to another. The easiest way to see the difference is to count people, not percentages.
“In plain words” is hidden on all puzzles.
Imagine the police took blood from the whole city. The thief and about 50 innocent people would match. Our defendant is one of these 51, and nothing else is known about him. So the chance that he is the thief is 1 in 51, about 2%. This card does not explain why "1 in 10,000" sounds like near proof, or where exactly the prosecutor goes wrong. The next one does.
The two questions sound almost the same, but their answers are very different. Compare: almost every pope is a Catholic, but almost no Catholic is the pope. "If pope, then Catholic" is nearly certain. "If Catholic, then pope" is one in over a billion. The prosecutor took the answer to the first question and passed it off as the answer to the second. Hence the name of the fallacy.
“Rigorous” is hidden on all puzzles.
1 : 499 999 × 10 000 = 10 000 : 499 999 ≈ 1 : 50
Before the test the odds that this man is the thief are 1 to 499,999: he is one of 500,000. The match is 10,000 times more likely if he is guilty (then it is certain) than if he is innocent (1 in 10,000). This number is called the likelihood ratio, and it multiplies the odds. The result is 1 to 50: a chance of guilt of 1/51 ≈ 2%. The match is strong evidence: it raised the odds 10,000-fold. But it all started from very low odds.
The same match, the same expert, the same figure of 1 in 10,000. On an island of 100 people it nearly proves guilt; in a country of 5 million it counts for almost nothing. The strength of the evidence depends not only on how rare it is but also on how many people could have left it. That is why a court needs to know how the suspect was found and whether there is other evidence.
G means the defendant is guilty, M means the trace matched. The expert gave P(M | not G) = p, while the court needs P(G | M). Bayes' theorem links them through the chance before the test, P(G).
The thief always matches: P(M | G) = 1. Before the test the defendant is one of N people: P(G) = 1/N. Substitute and cancel 1/N.
(N − 1)p is the expected number of innocent people who match. In our case N = 500,000, p = 1/10,000.
The same in odds form: odds after the evidence = odds before × the likelihood ratio.
The prosecutor's fallacy is replacing P(G | M) with 1 − P(M | not G), that is 1 − 1/10,000 = 99.99%. These numbers agree only when the chance before the test is around 50-50, and in a case where the defendant was found by the match alone it is very small.
“Deeper” is hidden on all puzzles.
In 1996 and 1998 Sally Clark's two baby sons, Christopher and Harry, died suddenly. In November 1999 she was convicted of murdering both and sentenced to life. Part of the evidence was the testimony of the paediatrician Roy Meadow: the chance of two cases of sudden infant death syndrome in a family like the Clarks' was 1 in 73 million.
The first mistake is multiplying dependent events. Meadow took an estimate of 1 in 8,543 for one such death in a similar family and squared it: 8,543² = 72,982,849. That is only valid for independent events. In October 2001 the Royal Statistical Society said publicly that the figure had no statistical basis, and in January 2002 its president, Peter Green, wrote to the Lord Chancellor that there were very strong reasons to think families may share genetic or environmental factors that make a second such death much more likely.
The second mistake is the prosecutor's fallacy. Even a correct figure is easily read as "the chance that the deaths were natural". Some press reports presented it that way. In the same letter the Society explained that a double murder of infants is also extremely rare, and that a court must compare how likely the deaths are under each of the two explanations, not just how unlikely they are under one.
The first appeal was dismissed in 2000. In January 2003 the conviction was quashed: it emerged that the prosecution pathologist had not disclosed microbiology results pointing to an infection in Harry. Sally Clark was released, and she died in 2007.
In Los Angeles witnesses described a couple of robbers: a blonde woman with a ponytail, a Black man with a beard and moustache, a yellow car. The prosecution gave each feature a frequency (from 1 in 3 to 1 in 1,000), a mathematician multiplied them and got 1 in 12 million. The Collinses, a couple who fit the description, were convicted.
In 1968 the Supreme Court of California reversed the conviction. The frequencies had no basis, and the features were not independent (beards and moustaches often go together). And even taking the prosecution's numbers as given, the court worked out in an appendix that the chance of at least one more such couple in the area was over 40%. Rare does not mean unique.
In 1987 the psychologists William Thompson and Edward Schumann named both fallacies. The defence attorney's fallacy is the opposite: "If the trace occurs in one person in 400, there are thousands of such people in our city, so the match means nothing." In fact the match narrows the field from millions to thousands, and together with other evidence it can make guilt very likely. Evidence should be neither overrated nor thrown out: it should be multiplied by everything else that is known.
A similar dispute is remembered from the O. J. Simpson trial (1995). The defence said that only a small share of men who beat their wives go on to kill them. The statistician I. J. Good replied in Nature: the right question is different — what share of murdered women who had been beaten by their husbands were killed by them. By Gerd Gigerenzer's estimate, about 8 in 9.
Mathematics: each piece of evidence multiplies the odds by its likelihood ratio. Logic: a rare event that has already happened proves nothing by itself: what matters is how well each explanation accounts for it. Practice: when a match is searched for among millions of profiles, chance matches become ordinary.
Mathematics
A city of 500,000 adults. The frequencies in this example are made up to show the mechanics.
For each piece of evidence ask: how many times more likely is it if the defendant is guilty than if innocent? A 1 in 10,000 blood match gives 10,000. A shoe print of a size one person in twenty wears gives 20. Evidence in the defendant's favour has a ratio below 1 and lowers the odds. Multiply the odds before the evidence by all these numbers in turn. Logarithms turn multiplication into addition, which is why the weight of evidence is sometimes counted in "points" that simply add up.
But only independent evidence may be multiplied. This is exactly where Meadow went wrong: two deaths in one family are not independent, and squaring overstated the rarity many times over. When pieces of evidence are linked (a beard and a moustache, a blood type and family ties), the second ratio has to be worked out given the first.
Logic
The couple's features occur in one couple in 12 million. One such couple certainly exists: the witnesses saw it. What is the chance there is at least one more in the area?
The figure "1 in 12 million" says how rare such a couple is. But the witnesses have already seen one, and now the question is different: is it the only one? If the area has millions of couples, another one turns up with a chance close to a half. The same goes for any rare event that has already happened: two infant deaths in one family, a series of deaths on one nurse's shifts, a lottery win. Its rarity proves nothing by itself. You have to compare how likely the event is under each explanation.
Practice
When a crime-scene trace is compared with a whole database rather than one suspect, every profile in the database is another chance for an accidental match. A partial profile that occurs in one person in 10 million gives at least one chance match in a database of a million people with a probability of about 9.5%. If a person was found exactly this way and there is no other evidence, "1 in 10 million" says much less about them than it seems. Check it yourself: grow the database or make the trace less rare.
Medicine uses the same logic: an accurate test for a rare disease raises many false alarms if everyone is tested. How to work it out for medical tests is shown in the false positive paradox. And after DNA cases where prosecutors made exactly this mistake (R v Deen, 1993; R v Doheny and Adams, 1996), the English Court of Appeal set out guidelines on how experts should present such evidence to juries.
The terms "prosecutor's fallacy" and "defence attorney's fallacy" were introduced by William Thompson and Edward Schumann in a 1987 paper in Law and Human Behavior. They studied how people interpret statistical evidence in court and found both mistakes among mock jurors.
It is 8,543 × 8,543: the estimate for one sudden infant death in a family like the Clarks' was simply squared, as if deaths in one family were independent. The Royal Statistical Society called the calculation invalid.
In 1968 the Supreme Court of California worked out that even with the prosecution's figures (1 couple in 12 million) the chance of at least one more such couple in the area was over 40%. The exact calculation for 12 million couples gives 41.8%.
In the Netherlands in 2003 the nurse Lucia de Berk was sentenced to life, partly because of a figure of 1 in 342 million: the supposed chance that so many deaths fell on her shifts by coincidence. Statisticians, among them Richard Gill, showed the calculation was wrong. She was acquitted in 2010.
For a 1 in 10,000 match to really point to one person in a city of 500,000, none of the other 499,999 could match. The chance of that is 0.9999 to the power 499,999, about 2 · 10⁻²². Practically zero.
In England the prosecutor's fallacy in DNA evidence has already led to successful appeals: R v Deen (1993) and R v Doheny and Adams (1996). The Royal Statistical Society pointed this out in its letter to the Lord Chancellor after the Sally Clark case.
Share your own explanation, an objection or a variant of the puzzle. You need a GitHub account to comment.
Is “School maths” the right level for this puzzle?
Thank you! This helps set the levels right.
I keep adding new puzzles, new ways to explain them and write-ups of the discussions they start. Leave your email and I'll write when something new is out. No spam, one-click unsubscribe.
The sign-up form will appear here soon.