A/B testing in plain words

"We redesigned the site on 1 November, and there were 30% more orders in November than in October. Isn't that proof?"

The simplest way to check a change is to compare the month before and the month after. In October an online shop got 4,500 orders. On 1 November a new design went live, and November brought 5,850 orders, 30% more.

From one month to the next everything changes at once: the season, the weather, ads, competitors' prices, the news. "Before and after" adds them all into one number and credits it to the new design. An A/B test does it differently: on the same days, half the people see the old site and half see the new one. The season, the weather and the ads are the same for both halves, so the difference between them comes from the design. The half with the old version is called the control group.

• "Before and after" shows everything that changed in that time, not just your change.
• An A/B test compares two versions on the same days, so everything else is the same for both.

How to judge a change when a test is impossible is the subject of the course "Does the feature work?".

"The new cart is seen by those who clicked "Try the new version" themselves. They buy 2.5 times as often. Shall we switch it on for everyone?"

A shop gets 1,000 visitors a week. Among them are 200 regular customers who buy often and 800 newcomers who mostly just look. A "Try the new cart" button appeared on the site, and 100 people clicked it.

Of those who clicked, 13 in 100 bought. Among the other 900, only 5 in 100.

When people choose for themselves, different people end up in the groups, and the difference in purchases shows the people, not the versions. A coin knows nothing about people: regulars and newcomers land in A and B equally, with a little wobble. That is why in an A/B test each person's group is picked by chance, not by them and not by a manager.

• If people chose the version themselves, the difference shows who chose, not which version is better.
• Groups must be split by chance: a coin, a lottery, the random generator in the testing tool.

• Sampling bias

  A sample some people get into more easily than others, so it describes everyone badly.

"Pages A and B are identical. So where does a difference come from?"

The best way to understand a test is to run it on two identical pages. Such a check is called an A/A test. A coin splits the visitors, and both halves see the same page. On it, an average of 5 people in 100 buy.

Each group got 100 visitors.

Even when the versions are the same, the numbers in the groups differ: chance decides exactly who came and who bought. The fewer the people, the stronger this wobble. So a difference on a dashboard does not yet mean one version is better. The question is always the same: is this difference bigger than what chance alone gives?

• There is a difference even between identical versions. Before celebrating, ask whether A against A could have shown such a gap.
• Percentages on small groups mislead: 4 against 7 purchases is "+75%".

• Law of large numbers

  The more repetitions, the closer the share of an outcome gets to its true probability.

This builds on what you haven't done yet: chapter 3, “Can a difference come from nowhere?”

"We have a thousand visitors per version. Isn't that enough?"

Now let B really be better. On the old page A, 5 people in 100 buy; on the new page B, 6 in 100. For a business that is a lot: one extra purchase for every five.

Note that there are two different percentages here. "5 in 100" and "6 in 100" are conversion rates: 5% and 6% of visitors bought. But the dashboard will say "B +20%": how much bigger B's number is than A's, since 6 is a fifth more than 5. It is the same gap, one purchase per 100 people, yet it can sound like "+1%" or like "+20%". When you see a lift on a dashboard, ask: from what to what?

The test runs until each version has been seen by 1,000 people. Then we compare where there are more purchases.

A test sees a difference only when it is bigger than the wobble of chance. And the wobble fades slowly: to make it half as big you need four times as many people. Hence a simple rule: the smaller the difference you want to notice, the more people you need. Half the difference means four times the people.

Exactly how many is what sample-size calculators work out, and there are free ones. They need two things: the current conversion rate and the smallest difference you want to notice. For 5 vs 6 in 100 it comes to about 6,500 people per version, six and a half times as many as in our example.

• Work out how many people you need before the test starts, not after.
• Next to "+20%" on a dashboard, look for from what to what: from 5 to 6 in 100 is just one purchase per 100 people.
• A small difference needs many people. If you don't have that many, test bolder changes or run longer.

This builds on what you haven't done yet: chapter 3, “Can a difference come from nowhere?”

"The dashboard says: B is ahead, confidence 95%. Is that a guarantee?"

The testing tool knows how the numbers wobble (chapter 3) and compares B's lead with that wobble. The bigger the lead and the more people, the higher the "confidence". A winner is usually declared once it reaches 95%.

To see what this threshold promises, take identical pages again, 1,000 people on each.

The easiest way to picture the 95% threshold is as the setting of a smoke alarm. You can make it so sensitive that it hears every spark, but then it will beep at every burnt piece of toast. 95% is an agreement: at "toast", meaning chance alone, the test goes off falsely about 1 time in 20. And a real "fire", a real difference, it notices more often the bigger it is and the more people are in the test.

This is the alarm the calculator from chapter 4 sizes the test for. 6,500 per version is enough for a real gap of 5 vs 6 in 100 to reach 95% in about 8 tests out of 10.

The 95% threshold protects against chance, but only up to one in twenty. And the promise holds only if you play fair: the number of people is set in advance, the result is read once at the end, and one version is checked against one. The next three chapters are three ways to break that promise without noticing.

One more thing: 95% says nothing about the size of the gain. "B +27%, confidence 95%" does not promise +27% after launch (chapter 8). Some tools show a "chance to beat" instead of confidence; it is computed differently, and a chapter of the course for marketers covers it.

• "95% confidence" means chance gives such a lead less than 1 time in 20. It is neither a guarantee nor the size of the gain.
• The promise holds only if the number of people was set in advance and the result is read once, at the end.

This builds on what you haven't done yet: chapter 5, “What does "95% confidence" mean?”

"The test is planned for 14 days, but on day three it is already at 95%. Why wait?"

The test is planned for 14 days, with 300 people seeing each version every day. The dashboard updates every morning, and every morning someone opens it. The agreement: as soon as confidence reaches 95%, the test stops and the winner is announced.

Take identical versions again, where there can be no winner.

The 95% threshold is designed for one look at the end. Every daily check with the right to stop is one more try, and fake winners pile up. You may look at the dashboard: to spot a bug or a disaster. You may not stop the test because it has "already gone green".

• Write down in advance how many people and days the test will take, and read the result only then.
• Test in whole weeks: different people buy on Monday and on Saturday.
• If you really need to stop early, there are special methods that correct for peeking. The ordinary "95%" is not one of them.

This builds on what you haven't done yet: chapter 5, “What does "95% confidence" mean?”

"AI wrote 10 headlines. We test them all against the old one, and one won with 95% confidence. Do we take it?"

The new headlines are no better than the old one: on each, the same 5 people in 100 buy. Each headline is seen by 2,000 people, the old one by 2,000 as well, and each new one is compared with the old.

The same happens when one test's result is cut into pieces: mobile and desktop, new and returning, cities, days of the week. Each piece is one more try, and among a dozen pieces "B wins on Android in Kharkiv" is easy to find. The more tries, the surer one of them wins by luck.

• The more variants or segments, the more fake winners. Count how many tries there were.
• Check a winner picked from many in a separate test: it against the old one, one on one.
• Write down the segments you care about before the start, rather than searching after.

This builds on what you haven't done yet: chapter 5, “What does "95% confidence" mean?”

"In the test B gave +20%, and after launch only +10%. Did someone make a mistake?"

This time B really is 10% better: 50 people in 1,000 buy on A, and 55 on B. Each version is seen by 5,000 people, and the test is fair: the number of people was set in advance and the result is read once at the end. The team runs many such tests and ships those where B won with 95% confidence.

This is called the winner's curse: when you pick the best, you also pick the lucky ones. The fewer people in a test, the bigger the push needed to win, and the more the winner exaggerates.

There is a second reason, with no chance involved: the novelty effect. A new bright button draws regular customers' attention because they haven't seen it before. In a few weeks they get used to it, and the difference fades. So a test that ran only a few days can show an effect that will be gone in a month.

• A winner's gain in the test is usually inflated. Plan budgets and forecasts on a smaller number.
• The more people in the test, the smaller the exaggeration.
• Run a test of a new, eye-catching change for at least two weeks, so the novelty has time to wear off.
• To see the real effect, keep a small group without the change for a few weeks after launch.

This builds on what you haven't done yet: chapter 4, “How many people do you need?”chapter 5, “What does "95% confidence" mean?”

"The test is over, and the dashboard is grey: B +4%, confidence 70%. So B doesn't work?"

The most common result of an A/B test is neither a win nor a loss but a grey "no winner". Here is such a test. It was sized honestly with the calculator from chapter 4: 6,500 people per version, to notice a gap of 5 vs 6 in 100. The test ran to its end, and the result was read once.

A grey result is neither a failure nor wasted work. It says something useful: the big difference the test was sized for is most likely not there. A small one may well be, it just needs far more people.

• Write it down honestly: "no big difference; the test couldn't have noticed a small one", not "B doesn't work".
• Decide on other grounds. If B is simpler, cheaper or needed for another reason and doesn't look worse, you can keep it. If B costs more, keep A.
• Don't extend the test until it goes green: that is the peeking from chapter 6.
• If even a small difference matters to the business, plan a new, bigger test, sized in advance. Or test a bolder change that is easier to notice.

This builds on what you haven't done yet: chapter 1, “Why test if there is "before and after"?”chapter 2, “Why does a coin split the groups?”chapter 3, “Can a difference come from nowhere?”chapter 6, “When may you stop a test?”chapter 7, “Why not test 10 variants at once?”chapter 9, “No winner. Now what?”

"Here are seven results. What do we do with them?"

Six dashboards hold one trap or question from the course each, and one is clean. Choose what to do. After you answer, an explanation appears with a link to the chapter that covered it.

Nine questions, one from each chapter. You can ask them of any dashboard without calculating anything yourself.

Ready for harder cases: revenue instead of purchases, a Bayesian "chance to beat", a test plan and a report for the boss? Next comes the course "A/B testing paradoxes", and you can check your own numbers with the "Check your test" tool.