Glossary / Holdout group
Over a year a big product team shipped 20 changes, and each one honestly won its A/B test. The analyst adds up the recorded lifts, and the report promises almost +50% in sales. Meanwhile 5% of customers spent the whole year on the old site and saw none of these changes. Surely they should buy noticeably less than everyone else. The comparison with them is the year's real check, and the group itself is called a holdout group.
“In plain words” is hidden on all puzzles.
The shop checks ideas with A/B tests of 400,000 visitors per variant, at a 5% conversion rate. Ideas vary: some help a little, others change nothing, a few do harm. Every idea that wins its test with a significant result is shipped, and its lift from the dashboard goes into the annual report. Over the year there are 20 such winners.
The report: 20 winners, about +49% together. What will the comparison with the holdout show at the end of the year?
lift recorded in the reportthe idea's true effect
The simplest explanation, slightly simplified: each test sees its change alone, for a few weeks and with noise, while the holdout sees all the changes together, for the whole year, with no picking of the best. So the report adds up optimistic pieces, and the holdout measures the whole picture. The gap between them comes from three separate reasons, and none of them depends on the analyst's honesty.
Move the slider to the right: with big tests the noise is smaller, and the first reason almost disappears. But even with 6.4 million visitors per variant the report promises +19% on average, while the holdout shows about +7%: no test size lets a single test see overlap and fading.
A holdout group, or long-term holdout, is a small random share of customers who for a long time, usually a quarter or a year, see none of the shipped changes or campaigns. Everyone else sees everything. The difference between them at the end shows what all the changes gave together, overlap and fading included. In CRM marketing such a group is often called a universal control group: these customers get no campaigns at all. And when part of the users are kept without one new feature after its launch, people call it a holdback.
An ordinary control group in an A/B test answers "is this change better than the old version right now". A holdout answers a different question: "how much better is our whole year of work than a year without it".
“Rigorous” is hidden on all puzzles.
A holdout has a price: these customers go a whole year without improvements. And it has a precision: the smaller the group, the noisier its result. Say the shop has 2 million customers a year, and 30% of them buy. The true gain from all changes is +7.1%, as in the simulation. Change the holdout's share.
Even 1% is enough to tell +7% from the promised +49%: the margin is then about ±2.1 points. To measure the gain itself to half a point, you already need a fifth of the customers. Halving the margin takes a group four times bigger, which is why people usually keep 5–10%: at 5% the margin is about ±1.0 point, and the price is roughly 0.36% of annual sales.
How recorded wins add up in an annual report step by step is covered in chapter 17 of the course "A/B testing paradoxes", and how to size a long-term holdout for a single feature in the course on feature impact.
“Deeper” is hidden on all puzzles.
The winner's curse. In the simulation the true effects of ideas are spread around zero with a standard deviation τ = 0.75%, and the noise of a test with 400,000 visitors per variant is σ ≈ 0.97%. If a test recorded a lift x, the best estimate of the true effect is only a fraction of it:
E[true | recorded x] = x · τ² / (τ² + σ²) = x · 0.56 / (0.56 + 0.95) ≈ 0.37 · x
So of the recorded +49% only about +18% remains, and 7% of the winners actually do harm. The bigger the test, the closer the factor gets to one: at 6.4 million per variant it is 0.90.
Overlap. In the model all changes together can add at most H = 15%: that is how many buyers can be persuaded at all. Each change persuades a share e/H of those still left. Then together:
together = H · (1 − (1 − e₁/H)(1 − e₂/H)…(1 − e₂₀/H))
For small effects this is almost the sum, but twenty changes of 0.9% each already give +10.7% instead of +18.2%. The limit H in the simulation is an assumption, and every business has its own. But the formula shows the direction: the more changes aim at one and the same purchase, the less their sum looks like the truth. If each test runs on top of the changes already shipped, it measures only its own addition, and such additions add up honestly. Overlap is counted twice where tests run in parallel, each against the old version.
Fading. The "two thirds remain" factor is also a model assumption. Only a long measurement shows the real share, and that is exactly what a holdout is kept for.
| Visitors per variant | Ideas for 20 wins | Report | True, alone | Together | Holdout |
|---|
An exact calculation under the normal approximation, the average for a year. The simulation above gives slightly different numbers every time: it is one year, not the average.
In 2018 Minyong Lee and Milan Shen of Airbnb published a KDD paper, "Winner's Curse: Bias Estimation for Total Effects of Features in Online Controlled Experiments". They showed that the sum of effects of launched experiments, chosen for being significant, systematically overstates the true total effect, and proposed a correction for this bias.
Fading and long-term effects were the subject of a 2015 paper by Henning Hohnhold, Deirdre O'Brien and Diane Tang of Google, "Focusing on the Long-term: It's Good for Users and Business". On search ads they showed that a short test does not always predict the long-term effect: users learn over time and change their behaviour. To see it, groups of people lived with different settings for a long time.
Long-term holdouts and holdbacks as a tool of their own are described in Ron Kohavi, Diane Tang and Ya Xu's book "Trustworthy Online Controlled Experiments" (Cambridge University Press, 2020), in the chapter on measuring long-term effects (p. 175).
A sample some people get into more easily than others, so it describes everyone badly.
In 1936 a magazine polled millions of Americans and still picked the wrong election winner.
The more repetitions, the closer the share of an outcome gets to its true probability.
The share of heads creeps towards 50%, while the gap between heads and tails in coins usually grows.
A number from −1 to +1 showing how closely two quantities move together.
In summer both ice cream sales and drownings go up. Ice cream isn't to blame: the heat drives both.
How far an average or a rate computed from a sample usually misses the truth: σ divided by the square root of the sample size.
Halving the error takes four times as much data.
A range around a measured number, built by a method that covers the true value in a chosen share of samples, such as 95%.
Of twenty honest 95% intervals, on average one misses the truth, and none of them knows whether it is the one.
How often chance alone would produce a gap at least as big as yours if there were really no difference.
When there is no effect at all, p below 0.05 turns up as often as p between 0.50 and 0.55.
The number on Bayesian dashboards: how likely B is to be better than A, given the data and a starting assumption.
"97% chance" tells the truth only when every idea of yours changes something; if four in five are useless, B is better in only a third of such tests.
The probability that a test notices a real effect of a given size. A power of 80% means the test misses one working effect in five.
A weak A/B test that does find an effect almost always exaggerates it: a real +10% shows up on the dashboard as +24%.
The smallest real effect a test of a given size will find with 80% probability (usually at a 5% significance level). Smaller effects the test will mostly miss.
A shop with 1,000 visitors a day would need seven years of testing to reliably notice a +2% lift in conversion.
A way to get the error of any number without a formula: redraw a sample of the same size from your own data with replacement many times and watch how the result jumps.
Each resample loses a third of your data and repeats the rest, yet the spread of such copies shows the real error.