Glossary / Bootstrap
You have 50 orders and you have computed the average order value. How far can you trust it? The honest answer is to collect a thousand more such weeks and watch how the number jumps. The bootstrap does this without a single new order: a thousand times it draws 50 orders from your own 50, putting each one back. Every such copy loses about a third of your data and repeats some of the rest. And yet the spread of these "damaged" copies is almost the same as that of real new weeks.
“In plain words” is hidden on all puzzles.
A resample works like this: 50 times we pick one of your 50 orders at random, write it down and put it back. So the same order can come up twice or three times, and another one not at all.
What share of your 50 orders usually doesn't make it into a resample even once?
36.4% of the orders go missing, usually about 18 of 50. For large samples the share tends to 1/e ≈ 36.8%.
Here is one resample. Each square is one of your orders, and the number in it is how many times it was drawn. Empty dashed squares didn't make it at all; dark ones came up twice or more.
In the simulation we know the whole shop: the true average order is $40, and single orders mostly run from $10 to $100, occasionally more. You see only 50 orders from one week. Press "Resample ×1000": the dark pile shows the averages of a thousand resamples. The grey pile is what you never get to see in real life: a thousand genuinely new weeks of 50 orders. Compare their widths.
1,000 resamples1,000 real new weeksyour weekthe truth
Notice that the dark pile sits around your week, not around the truth. The bootstrap doesn't tell you where the true average is. It tells you how far your number could have jumped, and that is exactly what the "±" in a report and a confidence interval need. For a median or a return rate there is no handy formula at all, so the bootstrap is often the only simple way to get the error.
An explanation without formulas, slightly simplified. A new week in the real shop means 50 new customers drawn at random from everyone. You can't visit everyone, but your 50 orders are the best portrait of the shop you have: small and large orders appear there in roughly the right proportion. A resample "visits" this portrait the way you would visit the shop. That some orders are missing and others repeat is not a flaw but the whole point: a new week would also miss some customers and bring several similar ones. That randomness is the very error we are after.
So the bootstrap is only as good as your portrait. If not a single big client happened to land among your 50 orders, the resamples won't learn about them either. And if the sample is biased, say you only surveyed people who answer the phone, the bootstrap will carefully measure the error of a biased number.
“Rigorous” is hidden on all puzzles.
The bootstrap is a way to estimate the error of any number computed from a sample, without a formula: you repeatedly draw a new sample of the same size from the sample itself, with replacement, and watch how the result changes. The steps:
Share left out: (1 − 1/n)ⁿ → 1/e ≈ 0,368 · for the mean: SE* = σ̂ / √n = √((n − 1)/n) · s / √n
Each order is missed on a single draw with probability 1 − 1/n, and over n draws with probability (1 − 1/n)ⁿ. For the mean, the bootstrap reproduces the known formula almost exactly: the spread of the resamples equals σ̂ / √n, where σ̂ is the spread of your n values. It differs from the usual s / √n by a factor of √((n − 1)/n), which is 0.99 for 50 orders. In other words, the bootstrap "invented" the standard error formula on its own, knowing nothing about it.
| Sample n | Left out | Distinct values | Record kept |
|---|
"Left out" is the share of values missing from a resample; "Distinct values" is how many different values remain in it; "Record kept" is the chance that the sample's largest value made it into the resample.
“Deeper” is hidden on all puzzles.
Switch the simulation to "Largest order". The dark pile turns into a single bar: in 63.6% of resamples the largest order is the same as in your week, because your record makes it into a resample with probability 1 − (49/50)⁵⁰. A resample can never go above it: it only knows the numbers you have already seen. Yet the real record of a new week beats yours in exactly half of cases: which of two weeks holds the bigger record is down to chance. In a typical week the spread of the resamples for the record is about three times smaller than the real one.
The same goes for anything that depends on the edges: the maximum, the minimum, the rarest event. The bootstrap works for numbers that rest on many values at once, like the mean or the median, and fails for those that rest on a single one.
A sample some people get into more easily than others, so it describes everyone badly.
In 1936 a magazine polled millions of Americans and still picked the wrong election winner.
The more repetitions, the closer the share of an outcome gets to its true probability.
The share of heads creeps towards 50%, while the gap between heads and tails in coins usually grows.
A number from −1 to +1 showing how closely two quantities move together.
In summer both ice cream sales and drownings go up. Ice cream isn't to blame: the heat drives both.
How far an average or a rate computed from a sample usually misses the truth: σ divided by the square root of the sample size.
Halving the error takes four times as much data.
A range around a measured number, built by a method that covers the true value in a chosen share of samples, such as 95%.
Of twenty honest 95% intervals, on average one misses the truth, and none of them knows whether it is the one.
How often chance alone would produce a gap at least as big as yours if there were really no difference.
When there is no effect at all, p below 0.05 turns up as often as p between 0.50 and 0.55.
The number on Bayesian dashboards: how likely B is to be better than A, given the data and a starting assumption.
"97% chance" tells the truth only when every idea of yours changes something; if four in five are useless, B is better in only a third of such tests.
The probability that a test notices a real effect of a given size. A power of 80% means the test misses one working effect in five.
A weak A/B test that does find an effect almost always exaggerates it: a real +10% shows up on the dashboard as +24%.
The smallest real effect a test of a given size will find with 80% probability (usually at a 5% significance level). Smaller effects the test will mostly miss.
A shop with 1,000 visitors a day would need seven years of testing to reliably notice a +2% lift in conversion.
A small random share of customers who for a long time see none of the shipped changes, so that comparing with them shows what all the changes gave together.
Twenty changes a year, each one winning its A/B test, almost +50% together, yet customers who saw none of them buy only about 7% less.