Courses / A/B testing paradoxes
Anyone who has run an A/B test has seen a line like this. Ship B? The course follows the questions that come next: isn't it noise, how many people do you need, can you stop early, what to do with a win in one segment. Five cases show traps the dashboard does not notice. A checklist closes it. Formulas are kept to a minimum, and every named term is first shown in a picture.
"B +17%. Is that a real effect, or just luck?"
Each visitor either buys or not, a bit like a coin. So even two identical pages never give identical numbers. Below that is exactly the case: A and B are the same, both truly convert at 5%, and each gets 1,000 people. Run a few such tests and watch where the lift jumps.
The typical size of this wobble is called the standard error: how far a number from a sample usually lands from the true one. Run hundreds of A/A tests and the lifts pile up into a bell: the normal distribution. The more visitors, the narrower the bell. Notice that lifts of tens of percent are ordinary here, even though the pages do not differ at all.
The same page is shown as "A" and as "B", converting at 5%, with 1,000 visitors each. How often will one of two identical pages "win" by 20% or more?
"Fine, noise happens. How many visitors do I need not to be fooled by it? And how do I check my result?"
The size of a test depends on three things: the usual conversion rate, the smallest effect that matters to you, and how strictly you judge. The industry standard: at most 5% false winners between identical pages and 80% power, meaning the test spots a real effect of the chosen size 4 times out of 5. A small effect hides in the noise, so it needs many people: a lift half as big needs about four times the sample.
Once a test has its sample, the result has to be checked. The check answers one question: if the pages were identical, how often would noise produce a gap at least as big as yours? That share is the p-value. If it is below 5%, the result is called significant. The confidence interval shows something else: all the effects your data are still compatible with. The numbers from the dashboard at the top of the page are already filled in below.
"Confidence 94%" in many tools means 1 minus the p-value. It is not the probability that B is better. It only measures how rarely noise produces a gap like this. Distributions, power and the winner's curse are covered in more depth in round 4 of the A/A test puzzle and round 5, with a calculator and a traffic split check.
How often chance alone would produce a gap at least as big as yours if there were really no difference.
A range around a measured number, built by a method that covers the true value in a chosen share of samples, such as 95%.
The probability that a test notices a real effect of a given size. A power of 80% means the test misses one working effect in five.
"The test is planned for a month, but on day nine the dashboard already shows 95%. Why wait?"
The dashboard updates every evening, and every evening you want a look. The 95% threshold is designed for one look at the end of the test. What happens when you look every day and stop at the first "95%" is shown in the puzzle below: the two pages are identical, and the marketer checks every evening for 30 days.
Two identical pages, a 30-day test, a check every evening and a stop at the first "95%". How often is one of the identical pages declared the winner?
"The calculator says 5 weeks. I don't have that long. What now?"
There is no free way: each option below pays with something. The numbers follow the calculator example: 5% conversion, we want to detect +10%, the 95% standard and 80% power, that is 31,234 visitors per page. Each option is worked through with an interactive in round 5 of the peeking puzzle.
"Overall B did nothing. But in Poland, on mobile, among new customers, it is significantly better. Ship it there?"
After a test that showed nothing, it is tempting to slice the result: countries, devices, traffic sources, new and returning customers. Each slice is a separate check, and each has its own chance of a false winner. The puzzle below works out what happens when a new email is no better than the old one and the result is split into 20 countries.
The new email is no better than the old one. The analyst checks the result in 20 countries. What is the chance at least one shows a "significant" difference?
"We started B on 10% of traffic and added more every day. B is better on every day of the week. So it is better for the week too?"
A gradual rollout sounds sensible: if B is broken, few people suffer. But it quietly changes which days end up in each group. A shop gets 10,000 visitors a day. On Monday 10% of them go to B, the share grows every day, and by Sunday it is half. People here buy half as often on weekends as on weekdays. And every day, B converts 5% better than A.
The classic version of this trap is in the catalogue. If you want to solve it yourself, do it before the case.
Treatment A beats treatment B for small kidney stones and for large ones. Can B be better for all the patients together?
Who won the week, if you add all visitors together?
ABcircle size = how many people saw the variant that day
"A new product card: more people add to cart, but fewer buy from the cart. Cart conversion fell from 40% to 33%. Is B worse?"
Visitors differ. Some came to buy, some are comparing, some are just browsing. The new card does not change those who were ready anyway. It persuades some of the lukewarm to put the item in the cart. These new people buy less often than those who added to cart before, so the share of carts that end in a purchase falls. Each dot below stands for 10 visitors.
hot, 20%: came to buywarm, 30%: comparinglukewarm, 50%: browsing
People are split at random only at the entrance: each visitor has an equal chance of seeing A or B. So A and B are fairly comparable only at the level of all visitors. Those who reached the cart have already been selected by the page itself, and in B they are different people than in A.
In town, looks and kindness are unrelated. You only date people who are good-looking and kind enough together. Are the two linked among your dates?
Bombers come back with holes in the fuselage and wings, and hardly any on the engines. Where should the armour go?
"After a week of testing the new banner gave +15%. We shipped it, and a month later revenue is the same as before. Where did the effect go?"
Returning customers notice something new and click out of curiosity. After a few days they get used to it, and the effect fades. Below is a model of such a test: the banner's true long-run effect is +2%, but on the first day novelty adds another +30%, and that extra halves every two days. Each variant gets 5,000 people a day, A converts at 5%. Choose the day you stop the test.
lift on a single daycumulative lift, as on the dashboardthe model's true daily effectwhat stays for good
Even a month does not fully save you: the cumulative lift averages all days, and the first week pulls it up. It is more honest to look at the lift in the last weeks, or at the lift by days since a person's first visit: if it is large in the first days of acquaintance and then melts, that is novelty. The reverse happens too: people get used to an unfamiliar interface, and the effect grows over time.
After praise players do worse, and after criticism they do better. Does that mean criticism works?
"Over the year we shipped every significant winner. In the report they add up to a big gain. Why has revenue barely moved?"
A team runs 40 tests a year. Only some of the ideas really work, and their effect is modest: +5% on a 5% conversion rate. The other ideas change nothing. Every significant winner goes into the product, and its lift from the dashboard is written into the report. Assume also that half a year later a third of a real effect has faded, as in chapter 8. Compare three numbers: what was written in the report, what was real, and what a holdout group that saw the old site all year will show.
shipped, the idea really worksshipped, the idea is emptynot shipped
Three mechanisms are at work here. First: false winners, empty ideas that won thanks to noise. Second: the winner's curse. A low-power test lets a real idea through only when noise happened to help it this time, so the recorded lift of a real winner is usually several times its true effect. Third: effects fade after the launch. The bigger the sample, the weaker the first two mechanisms: move the second slider to the right.
1% of people have a disease. The test is 99% accurate, and you test positive. What is the chance you're really ill?
"The tool splits people into A and B, and the report shows session conversion. What's the difference?"
One loyal customer comes twenty times a month, and those sessions are alike: if they like to buy, they buy often. For the test they are not 20 independent observations but roughly one. A check that counts every session separately believes it has far more data than it does, and so sees "significant" differences where there are none. Below is an A/A test: 1,000 people per variant, 90% of them visit once, and 10% are loyal customers who visit many times.
Each column is one person, each square a session, a filled one a session with a purchase. Loyal customers' purchases come in bunches.
z in the tests, counting sessions"significant": |z| > 1.96the curve the check assumes
How wrong the check is can be estimated with the simple design-effect formula: the noise is really 1 + (m − 1)·ρ times bigger, where m is the number of sessions per person (for unequal visits, the average weighted by sessions) and ρ is how alike one person's sessions are. When everyone visits once, or one person's sessions are independent, the formula gives 1 and there is no error.