The dashboard says "97% chance to beat". Is B really better in 97 such tests out of 100?

One set of data, two questions

This is a simplified explanation, but it holds the whole idea. The test: 5,000 visitors per button, 250 bought from A (5.00%), 293 from B (5.86%). B is ahead by 17%. The dashboard adds two numbers:

These questions cannot be swapped. If a person plays in the NBA, they are almost certainly taller than 190 cm. But if a person is taller than 190 cm, they almost certainly don't play in the NBA: there are millions of tall people and a few hundred NBA players. The answer to the second question depends on how many basketball players there are. In the same way, the chance that B is better depends on how often your ideas improve anything at all. The same trap with a medical test is the subject of the false positive puzzle.

How does the dashboard know how often your ideas work? It doesn't. It assumes any change is equally possible, +20% as well as −20%. The simulation shows what comes of that.

The paradox of two promises

Each idea here is an A/B test with 5,000 visitors per version and a 5% conversion rate for A. Some ideas change nothing; the rest change B's conversion by a random amount from −30% to +30%. The computer runs thousands of such tests and picks out the ones where the dashboard shows the same as ours: B ahead, confidence from 93% to 95%, chance to beat about 97%. Then it looks at the truth: is B really better? Start with a world where every idea changes something, then move the slider.

Both numbers promise something. Confidence: if an idea changes nothing, you will see 94% confidence or more in only 6 tests out of 100. This promise holds at any position of the slider, because it speaks only about useless ideas. Chance to beat: among tests with this dashboard, B is better in 97 out of 100. This promise holds only in the world the dashboard assumed, where every idea changes something. The more useless ideas there are, the more of the "97%" tests are ones where B is no better.

When to use which

• "How often will a useless idea pass the filter?" ConfidenceWhen there are many tests and it matters that duds rarely become "winners". The promise doesn't depend on which ideas you test. This is how science, medicine and regulators work.
• "How likely is it that B is better?" Chance to beatWhen you need to make one decision and explain it to people. Remember the assumption: the dashboard treats any change as equally possible.
• Know how often your ideas work? Use itIf experience says only one idea in five does anything, the dashboard's "97% chance" is too high. The calculator below computes the chance with your experience.
• "How much will I gain?" Neither of the twoBoth numbers speak about direction, not size: B can be better almost surely, but by pennies. The size is shown by the lift with its confidence interval and by the expected loss.
• In both cases, a plan in advanceBoth numbers are honest when you look at them once, on a day set in advance. What happens if you peek every day is the subject of the A/B test peeking puzzle.

Compute both numbers

Enter your test. The slider is your experience: what share of your team's ideas usually changes nothing. The rest, as in the simulation, change the conversion rate by an amount from −30% to +30%.

Why 97 sits next to 94

Confidence counts the gap in both directions: "6 tests out of 100" is about 3 where noise lifted B and 3 where it lifted A. Chance to beat asks about one side only: is B better. With large samples and a flat assumption this gives a simple rule:

chance to beat ≈ 100% − (100% − confidence) / 2  ·  100 − 6/2 = 97

Hence the threshold trap. "95% chance to beat" corresponds to only 90% confidence, and "95% confidence" to a 97.5% chance. A team that moves from one tool to another and keeps the 95% threshold quietly changes the rules of the game.

Both 99.9%, yet B gives almost nothing

A big site: 4.5 million visitors per version, A converts at 5.00%, B at 5.05%. Confidence 99.94%, chance to beat 99.97%. Both numbers shout "B won", and B really is almost surely better. But only by 1%: one extra purchase per 2,000 visitors, and the confidence interval of the lift runs from +0.4% to +1.6%. Both numbers measure how sure you are of the direction, not what B is worth. If the change costs anything, the size decides.

Lindley's paradox: a bigger test, weaker evidence

Take a world where half of the ideas change nothing and the rest are spread from −30% to +30%. And four tests, each with about 95% confidence, but of different sizes.

The bigger the test, the smaller the gap that gives the same 95%. With 5 million visitors it is +0.5%: such a crumb is far easier to explain by a useless idea that noise nudged a little than by an idea spread up to ±30%. The dashboard with its flat assumption doesn't see this; its chance is about 97.5% everywhere. This is Lindley's paradox: Dennis Lindley described it in 1957 in the journal Biometrika. At the same confidence and an ever larger sample, the Bayesian conclusion can be pulled as far from the frequentist one as you like. The exact numbers depend on the assumed spread: the wider the spread, the stronger the effect.

Two schools and one question about boys

Behind the two numbers stand two schools of statistics. The frequentist school treats only the data as random: there is some effect, we don't know it, but we assign it no probabilities. So it asks "how often would data like these appear if there were no effect" and answers with the p-value, confidence and the confidence interval. The Bayesian school lets you speak in probabilities about the effect itself, as long as you say honestly what you believed before the data. So it asks "how likely is the effect after these data" and answers with the chance to beat.

Both questions were put to the same kind of data: who is born more often, boys or girls. In 1710 John Arbuthnot asked in the frequentist way: if the chances were equal, how improbable is it that more boys than girls were christened 82 years in a row? Around 1781 Pierre-Simon Laplace asked in the Bayesian way. From the Paris records of 1745–1770 (251,527 boys and 241,945 girls) he computed how likely it is that the chance of a boy does not exceed ½. It came out at about 10⁻⁴²: the boys' chance to beat was practically 100%.

In the 20th century Ronald Fisher and Jerzy Neyman built frequentist statistics because the starting assumption seemed too subjective to them, while Harold Jeffreys defended the Bayesian approach in 1939. Bayesian calculations long stayed too heavy for practice, until the 1990s brought powerful computer methods. Today A/B tools show both, often on one screen. The mistake is not choosing a school, it is reading one school's number as the answer to the other's question.