Math paradoxes

Glossary / Holdout group

Glossary · holdout group

Twenty wins a year, so where is the growth?

Over a year a big product team shipped 20 changes, and each one honestly won its A/B test. The analyst adds up the recorded lifts, and the report promises almost +50% in sales. Meanwhile 5% of customers spent the whole year on the old site and saw none of these changes. Surely they should buy noticeably less than everyone else. The comparison with them is the year's real check, and the group itself is called a holdout group.

In plain wordspictures, analogies, small numbers

“In plain words” is hidden on all puzzles.

A year of wins

The shop checks ideas with A/B tests of 400,000 visitors per variant, at a 5% conversion rate. Ideas vary: some help a little, others change nothing, a few do harm. Every idea that wins its test with a significant result is shipped, and its lift from the dashboard goes into the annual report. Over the year there are 20 such winners.

The report: 20 winners, about +49% together. What will the comparison with the holdout show at the end of the year?

Where the percent went

The simplest explanation, slightly simplified: each test sees its change alone, for a few weeks and with noise, while the holdout sees all the changes together, for the whole year, with no picking of the best. So the report adds up optimistic pieces, and the holdout measures the whole picture. The gap between them comes from three separate reasons, and none of them depends on the analyst's honesty.

  • 1. The lucky ones winThe true effects of ideas are small, smaller than a test's noise. A significant win more often goes to the idea that noise happened to push up this time. So a winner's recorded lift is bigger than its true effect on average, and some "winners" actually do harm. This is the winner's curse.
  • 2. Effects overlapTwo abandoned-cart emails, each bringing back 30% of carts in its own test. Together they bring back not 60% but 51%: the second email can only bring back a cart the first one didn't. A buyer can be persuaded only once, yet each test claims them.
  • 3. Novelty fadesA test lasts a few weeks, and part of the effect comes simply from something being new. Half a year later people are used to it. In the simulation two thirds of the effect remain.

Move the slider to the right: with big tests the noise is smaller, and the first reason almost disappears. But even with 6.4 million visitors per variant the report promises +19% on average, while the holdout shows about +7%: no test size lets a single test see overlap and fading.

What a holdout group is

A holdout group, or long-term holdout, is a small random share of customers who for a long time, usually a quarter or a year, see none of the shipped changes or campaigns. Everyone else sees everything. The difference between them at the end shows what all the changes gave together, overlap and fading included. In CRM marketing such a group is often called a universal control group: these customers get no campaigns at all. And when part of the users are kept without one new feature after its launch, people call it a holdback.

An ordinary control group in an A/B test answers "is this change better than the old version right now". A holdout answers a different question: "how much better is our whole year of work than a year without it".

Rigoroustables, charts, formulas

“Rigorous” is hidden on all puzzles.

How many customers to keep out

A holdout has a price: these customers go a whole year without improvements. And it has a precision: the smaller the group, the noisier its result. Say the shop has 2 million customers a year, and 30% of them buy. The true gain from all changes is +7.1%, as in the simulation. Change the holdout's share.

People in the group–
Margin, 95%–
Price · share of annual sales–

Even 1% is enough to tell +7% from the promised +49%: the margin is then about ±2.1 points. To measure the gain itself to half a point, you already need a fifth of the customers. Halving the margin takes a group four times bigger, which is why people usually keep 5–10%: at 5% the margin is about ±1.0 point, and the price is roughly 0.36% of annual sales.

Rules for a holdout

  • A lottery, not convenienceThe group is drawn at random before the year starts. "Those who didn't update the app" or "those who unsubscribed" are not a holdout: these people differ on their own.
  • The same people all yearSplit by customer, not by visit. If a person drifts in and out of the group, they see half of the changes, and the comparison gets blurred.
  • No exceptionsThe holdout sees no new features, no new emails, no discounts. One "very important" campaign for everyone, and the group now measures a year without everything except that campaign.
  • One comparison at the endIf you search every week for a segment where the holdout "already shows" a difference, the same false discoveries it was meant to guard against come back.
  • Discount the report, don't add it upThe ratio "holdout / sum of lifts" for last year is an honest correction factor for next year's plans. In the simulation it is about 0.15.

How recorded wins add up in an annual report step by step is covered in chapter 17 of the course "A/B testing paradoxes", and how to size a long-term holdout for a single feature in the course on feature impact.

Deepergeneralisations, theory, history

“Deeper” is hidden on all puzzles.

The three corrections as formulas

The winner's curse. In the simulation the true effects of ideas are spread around zero with a standard deviation τ = 0.75%, and the noise of a test with 400,000 visitors per variant is σ ≈ 0.97%. If a test recorded a lift x, the best estimate of the true effect is only a fraction of it:

E[true | recorded x] = x · τ² / (τ² + σ²) = x · 0.56 / (0.56 + 0.95) ≈ 0.37 · x

So of the recorded +49% only about +18% remains, and 7% of the winners actually do harm. The bigger the test, the closer the factor gets to one: at 6.4 million per variant it is 0.90.

Overlap. In the model all changes together can add at most H = 15%: that is how many buyers can be persuaded at all. Each change persuades a share e/H of those still left. Then together:

together = H · (1 − (1 − e₁/H)(1 − e₂/H)…(1 − e₂₀/H))

For small effects this is almost the sum, but twenty changes of 0.9% each already give +10.7% instead of +18.2%. The limit H in the simulation is an assumption, and every business has its own. But the formula shows the direction: the more changes aim at one and the same purchase, the less their sum looks like the truth. If each test runs on top of the changes already shipped, it measures only its own addition, and such additions add up honestly. Overlap is counted twice where tests run in parallel, each against the old version.

Fading. The "two thirds remain" factor is also a model assumption. Only a long measurement shows the real share, and that is exactly what a holdout is kept for.

Visitors per variantIdeas for 20 winsReportTrue, aloneTogetherHoldout

An exact calculation under the normal approximation, the average for a year. The simulation above gives slightly different numbers every time: it is one year, not the average.

How we know

In 2018 Minyong Lee and Milan Shen of Airbnb published a KDD paper, "Winner's Curse: Bias Estimation for Total Effects of Features in Online Controlled Experiments". They showed that the sum of effects of launched experiments, chosen for being significant, systematically overstates the true total effect, and proposed a correction for this bias.

Fading and long-term effects were the subject of a 2015 paper by Henning Hohnhold, Deirdre O'Brien and Diane Tang of Google, "Focusing on the Long-term: It's Good for Users and Business". On search ads they showed that a short test does not always predict the long-term effect: users learn over time and change their behaviour. To see it, groups of people lived with different settings for a long time.

Long-term holdouts and holdbacks as a tool of their own are described in Ron Kohavi, Diane Tang and Ya Xu's book "Trustworthy Online Controlled Experiments" (Cambridge University Press, 2020), in the chapter on measuring long-term effects (p. 175).

Where it shows up

More on “Data and experiments”

All terms →