Planes coming back from combat have most of their holes in the fuselage and wings, and hardly any on the engines. Armour is heavy, so only two parts of the plane can get it. Which ones? Lock in your answer, then check it in a game, on a simulation of ten thousand sorties and with the explanations.
Planes coming back from combat have most of their holes in the fuselage and wings, and hardly any on the engines. The armour belongs exactly on the engines and the cockpit. There are few holes there not because nothing hits them, but because planes hit there do not come back. We only see the survivors. The idea is linked to Abraham Wald and his memoranda of 1943.
The Second World War. Bombers fly over enemy territory every day, and some of them do not come back. Command wants to add armour, but knows that a heavy plane is slower and carries fewer bombs.
Every plane that comes back is inspected, and every hole from bullets and shrapnel is marked on a diagram. After a few months there are lots of diagrams.
Out of every 100 holes on the returning planes: 42 in the fuselage, 40 in the wings, 10 in the tail, 4 on the engines and 4 in the cockpit.
There is only enough armour for two of the plane's five parts: the engines, the cockpit, the fuselage, the wings or the tail. Armour does not make a part invulnerable, but it lowers the chance that a hit there brings the plane down.
Where should the armour go so that as few planes as possible are lost?
"Where the holes are: on the fuselage and wings"
In fact the engines and the cockpit need the armour. Holes on the returning planes show hits the plane survived. Hits after which it did not come back are not on the diagrams at all.
There is armour for two parts. Pick an option before reading on.
A breakdown of your answer will appear here.
Hard to decide? Open the hints one at a time: the easy one first, then stronger ones.
Imagine that any hit on an engine downs the plane at once. How many engine holes would you find on the planes that come back?
Holes are counted only on the planes that came back. If bullets land evenly, the engines, which make up 12% of the area, should take about 12% of the hits. Where are the rest?
Compare holes not with each other but with area. A part with far fewer holes than its area suggests is where a hit stops planes from coming back.
Pick two parts to armour by clicking the plane or the buttons, and send 100 bombers on a mission. The diagram shows the holes on the planes that came back. Then show the planes that did not return and see where they were hit.
Each plane takes a random number of hits, 1.2 on average, and each hit lands at a random spot on the plane. A hit on an engine brings the plane down with probability 70%, on the cockpit 60%, on the tail 20%, on the wings 8%, on the fuselage 4%. Compare where all the planes were hit with where the holes are on those that came back. Change the number of hits and how vulnerable the engines are.
Bars show the simulation, the tick the exact value from the formula in round 4.
The hole diagrams come only from planes that came back. That is not a random sample of all planes: a plane gets into it only if it survived its hits.
Then the returning planes would have no engine holes at all. There are two explanations: either the enemy somehow hits everywhere except the engines, or every engine hit brings the plane down. The second is far more plausible.
Our case is milder, but the logic is the same. The engines make up 12% of the area, yet carry only 4% of the holes. The other engine hits stayed on the planes that never returned.
Say each of 1,000 planes was hit once. By area the engines should have taken 120 hits. The planes that came back carry just 36. So 84 engine hits went down with their planes.
| Part | Expected | Found | Missing |
|---|---|---|---|
| Engines | 120 | 36 | 84 |
| Cockpit | 80 | 32 | 48 |
| Fuselage | 350 | 336 | 14 |
| Wings | 350 | 322 | 28 |
| Tail | 100 | 80 | 20 |
| Total | 1 000 | 806 | 194 |
806 planes came back, each with one hole. 194 were lost, and 132 of them were downed by a hit on the engine or cockpit.
Every hole on a returning plane proves the plane survived that hit. A fuselage riddled with holes is the best evidence that the fuselage needs armour least.
Compare holes not with each other but with area. The fuselage has 1.19 times the average number of holes per unit of area, the engines just 0.37. The lower the ratio, the more often a hit there is fatal.
In his memoranda Wald spelt out assumptions like these carefully and checked how they affect the conclusion.
Abraham Wald worked in the Statistical Research Group (SRG) at Columbia University. In 1943 he wrote eight technical memoranda on how to estimate a plane's vulnerability from the damage on survivors. They contain no advice on where to put armour: decisions were for the military.
The armour story was retold by the group's director, Allen Wallis, in 1980: the military wanted to protect the parts with the most hits, while Wald, he wrote, "assumed, on good evidence, that hits in combat were uniformly distributed". The dramatic details that circulate online are mostly later reconstructions.
The famous plane covered in red dots is modern. Wikipedia user McGeddon drew it in 2016 over a drawing of the Lockheed PV-1 Ventura patrol plane, and in 2021 Martin Grandjean turned it into a vector image. The idea comes from later US Air Force "hit plots". No diagrams by Wald, if he ever drew any, survive.
A hit lands on part s with probability equal to its share of the area, as. The plane survives a hit there with probability 1 − ks, where ks is the part's vulnerability.
Bayes' formula: where the hole is on a plane that came back.
Holes relative to area: the further below 1, the more vulnerable the part. Hence an estimate of vulnerability when we know how many planes flew (N) and came back (S).
If a plane takes several hits, their number is random (Poisson with mean λ), and each hit is independent. Then the shares of holes on the survivors are the same whatever λ is. The share that returns, and λ estimated from the holes:
Wald's basic equation: Si is how many planes came back with i hits, qi the chance of surviving the i-th hit. In Wald's example (as given by Mangel and Samaniego) 380 of 400 planes came back, 320 of them unhit.
The first row sets up the model. The second is Bayes' formula: the share of holes on the engines drops from 12% to 4.5%. The third turns it around: from the holes found, Hs, and the area we estimate how deadly a hit on each part is. The fourth extends this to several hits. The last is the equation from Wald's memoranda: if the chance of surviving each hit is the same, in his example it is about 85%.
How to see the invisible from the survivors' data, why surviving funds look smarter than the market, and where else this mistake hides.
The computer has made up a new vulnerability for each part. You see only how many planes flew and came back, and where the holes are on the returning ones. The method from round 4 estimates the vulnerability, then you can reveal the truth.
| Part | Area | Holes | Estimate | Truth |
|---|
The dead cannot tell their story, but their absence can be measured. The number of holes in each part on the survivors, together with the share that came back, gives exactly as many equations as unknowns: the average number of hits and the vulnerability of each of the five parts.
The precision depends on how many planes flew. With 200 sorties the estimate for small parts such as the cockpit is often 10 to 30 points off. With 10,000 the error is just a few points. In his memoranda Wald worked with exactly this kind of uncertainty: he derived not a single number but bounds within which the answer lies.
Wald's basic equation solves an even harder problem: knowing only how many planes came back with 0, 1, 2… hits, estimate the chance of surviving each further hit. For the 400-plane example it comes to about 85% per hit.
1,000 funds with no skill at all: each year every one trails the market by 1 point on average (fees) plus random noise. A fund that falls behind the market by more than the threshold is shut. After 15 years we average the results.
Funds, like planes, do not "die" at random: the ones closed or merged are precisely those that earn badly. So the average record of the funds that exist today is inflated. In the simulation no fund can beat the market, yet the survivors beat it on average. All funds together, the closed ones included, trail it by about their fees.
In 1996 Elton, Gruber and Blake put this inflation for US equity funds at about 0.9 points a year. According to S&P's SPIVA report for 2020, nearly 70% of US domestic equity funds disappeared over 20 years. Those that remain are the ones in the advertised league tables.
In 1987 vets at the Animal Medical Center in New York described 132 cats that had fallen from windows: of those treated, 90% survived. The study became famous for the observation that cats falling from higher floors had, on average, fewer injuries than those falling from middle ones. The explanation offered was that the cat has time to relax and spread out.
But only cats brought to the clinic alive got into the study. A cat killed by the fall is less often taken to a vet. In 1996 The Straight Dope column pointed this out; the clinic's head disputed that it skewed the data. A Croatian study in 2004 (119 cats) found the opposite: falls from the seventh floor or higher went with more severe injuries. The "safe height" claim stands on weak ground.
Wald's eight memoranda of 1943 were not published during the war. Allen Wallis kept the group's papers in a box and around 1980 gave them to Phil DePoy at the Center for Naval Analyses (CNA). He kept only Wald's work and in July 1980 issued it as "A Method of Estimating Plane Vulnerability Based on Damage of Survivors".
Wald's memoranda are highly technical. In 1984 Marc Mangel and Francisco Samaniego explained them in "Abraham Wald's Work on Aircraft Survivability" in the Journal of the American Statistical Association. That is where the example comes from: 400 planes, 380 back, 320 without a single hit.
The picture attached to every retelling of the Wald story was drawn on Wikipedia in 2016 over a drawing of a Lockheed PV-1 Ventura. The mathematician Bill Casselman noted that the internet illustrates the story with any plane at all, down to a DC-3 transport.
That is how much, by Elton, Gruber and Blake's estimate (1996), the average return of US funds is inflated if you count only the survivors. Over 20 years a gap of 0.9 points a year compounds to almost a fifth of the capital.
That many cats that fell from a height were described in 1987 in the Journal of the American Veterinary Medical Association. 90% of them had chest injuries, and 90% of the treated cats survived. There were no dead cats in the sample.
Abraham Wald was born in Cluj in 1902. After Austria was annexed he emigrated to the US. He died in a plane crash in the mountains of southern India in 1950, aged 48, without ever writing a memoir of his wartime work.
Share your own explanation, an objection or a variant of the puzzle. You need a GitHub account to comment.
I keep adding new puzzles, new ways to explain them and write-ups of the discussions they start. Leave your email and I'll write when something new is out. No spam, one-click unsubscribe.
The sign-up form will appear here soon.