An online shop has 250 discount coupons and a model that knows perfectly which visitors are most likely to buy. The coupons go to the 250 best candidates. How many extra purchases does that bring compared with a day without coupons? Lock in your answer, then play against two bots, watch the simulation and read the explanations.
The likeliest buyers would buy without a discount, and some of them are put off by a coupon. So 250 coupons for the "best" visitors give −22.5 purchases, while the same coupons for the people they actually move give +61. The game, the simulation and the explanations below show how to tell the difference.
A discount coupon appears in a pop-up on the site. It nudges some people to buy, leaves others cold, and some, faced with yet another "great offer" window, simply close the tab.
A thousand people visit the site in a day, of four kinds. Nobody's kind is written on their forehead, but this is how they behave:
The percentages are the chance of buying. Sleeping dogs are people who find one more coupon pop-up annoying: they leave, or remember they have long meant to unsubscribe.
The analysts trained a "who will buy" model. It is perfect: for each visitor it knows exactly their chance of buying without a coupon. It doesn't see a person's kind, only that chance.
There are 250 coupons. They go to the 250 visitors with the highest chance of buying according to the model.
How many extra purchases do these 250 coupons bring compared with a day with no coupons at all?
"The model is perfect and the coupons go to the best customers. Sales are bound to go up"
In fact the 250 likeliest buyers are the 100 sure things (90%) and the 150 sleeping dogs (40%). The coupon does nothing for the first and cuts the second from 40% to 25%. In all, 150 × (−0.15) = −22.5 purchases.
With no coupons the shop gets 177 purchases a day. The coupons go to the 250 visitors the model ranks as the likeliest buyers. Compared with a day without coupons, the number of purchases will be:
Pick an answer. The exact calculation and a simulation of 2,000 shop days will appear here.
Hard to decide? Open the hints one at a time: the easy one first, then stronger ones.
A coupon is useful not where a person will buy, but where they will buy because of it. For each kind, compare the chance without a coupon and with one.
Who are these 250 likeliest buyers? Take the kinds in order of their chance of buying without a coupon and add people until you reach 250.
Work out what the coupon does to everyone in those 250: how many purchases are added or lost per hundred such people?
In front of you are 24 visitors of the same shop. You can't see their kind, only clues from their history, and the clues are noisy. Hand out the coupons, then open the parallel universe: each card shows what the person would do without a coupon and with one. Two bots play the same deal: the "who will buy" model gives coupons to the likeliest buyers, the uplift bot to those a coupon is most likely to move.
| Luck removed | Discounts to those who'd buy anyway |
|---|
The same shop with 1,000 visitors. The chart shows how many extra purchases the first hundred, two hundred and so on up to a thousand coupons bring, handed out by three rankings: the "who will buy" model, the uplift ranking (whom a coupon helps most) and at random. The lines are exact; the dots come from the simulation.
The "who will buy" model answers the question "what will this person do?". A coupon asks a different one: "what changes if I give them a coupon?". The answers differ, and are often opposite.
“In plain words” is hidden on all puzzles.
Someone who buys 90% of the time will buy without a discount. The coupon adds nothing; it only cuts revenue by the discount. The "who will buy" model puts such people first, because they really are the likeliest buyers. But a coupon is needed not by those who will buy, but by those who will buy because of the coupon. This card only explains why coupons are wasted; why purchases even go down is the next card.
After the 100 sure things the model takes the next highest chance: the sleeping dogs with their 40%. The persuadables at 10% and the lost causes at 2% look hopeless to the model, so their turn never comes.
For the sleeping dogs the coupon does harm: the chance drops from 40% to 25%. Of 150 such people, 60 would buy without a coupon and 37.5 with one. That is where −22.5 comes from. The saying "let sleeping dogs lie" means "leave well alone".
The model in the puzzle is perfect: every "who will buy" forecast it makes is right. A better model would not fix anything. The mistake lies in the question: picking people by their chance of buying when what matters is how much that chance changes.
The difference between "with a coupon" and "without" is called the uplift. For persuadables it is +40 points, for lost causes +1, for sure things 0, for sleeping dogs −15.
In the game you saw both worlds; in real life you never do: a person either got the coupon or didn't. So the uplift of one person can't be measured. For a group it can: some similar visitors get a coupon, some at random don't, and the difference between the groups shows what the coupon did.
“Rigorous” is hidden on all puzzles.
| Kind | People | No coupon | Coupon | Uplift per 100 coupons |
|---|
The model sorts by the "no coupon" column: 90%, 40%, 10%, 2%. The uplift ranking sorts by the last column: +40, +1, 0, −15. The model's first 250 coupons go to the sure things and the sleeping dogs, with uplifts 0 and −15. The uplift ranking's first 250 go to the 150 persuadables and 100 lost causes: 60 + 1 = +61.
The uplift is defined as the chance of buying with a coupon minus the chance without one. For any single person only one of the two parts is available. The way out is a randomised experiment: among visitors with the same features, half get a coupon at random and half don't. At random means the groups are on average the same in everything except the coupon, so the difference in their purchase rates is the uplift.
It is the same A/B test, only the result is split by people's features: whom the coupon helped, whom it didn't, whom it hurt. How much noise such a comparison carries is on the "Two identical pages" page. A "who will buy" model needs no control group, which is why it is built far more often.
| Clues | Share | Persuadables per 100 | Sleeping dogs per 100 | Buys without | Uplift per 100 coupons |
|---|
The eight most common sets of clues in the game. Among 100 visitors with the same set there are this many people of each kind, and the chance of buying and the uplift are averages over such a hundred. "Weekly" raises the chance of buying, but without "cart" it makes the uplift negative: many such people are sleeping dogs. The best sign for a coupon is "cart" with no other clue. The "who will buy" model takes the rows with the highest "buys without", the uplift bot those with the highest uplift.
Along the axis is the share of people who got a coupon; upwards is the total uplift in purchases. The "at random" line joins zero with the result of coupons for everyone. The higher a curve rises above that line, the better the model finds the people a coupon moves. The area between the curve and the line is called the Qini.
The uplift for people with features x: the difference of two chances that are never observed together for one person.
The "who will buy" model's 250 coupons: 100 people with uplift 0 and 150 with uplift −0.15.
The clues in the game: the probability of kind k given a set of clues c (Bayes' rule; each clue shows up independently with a probability that depends on the kind).
The Qini: the area between the uplift curve G and the random line, where x is the share of people with a coupon.
“Deeper” is hidden on all puzzles.
In 1986 the statistician Paul Holland named this obstacle in "Statistics and Causal Inference" (Journal of the American Statistical Association). To know what a coupon did to a particular person, you would need to see their outcome both with the coupon and without it, which is impossible: a person lives only one of the two. Holland called this the fundamental problem of causal inference.
He also set out the statistical solution: instead of the effect for one person, estimate the average effect for a group, by comparing groups formed at random. Uplift models do exactly that, only separately for each combination of features.
In 1999 Nicholas Radcliffe and Patrick Surry, at a credit scoring conference in Edinburgh, described "differential response analysis": modelling not the response to a mailing but the difference between the response of those who got it and of a control group. They later called it uplift modelling.
In 2002 Victor Lo, in SIGKDD Explorations, proposed the "true lift model": a single regression on the data of both groups, in which the "got the mailing" flag enters both on its own and multiplied by the other features. In 2007 Radcliffe described the incremental gains curve and the Qini measure for comparing such models.
What happens when you count money instead of purchases; why the same logic stands behind a control group; and where sleeping dogs bite in real companies.
| Coupons to | Coupons | Extra purchases | Profit |
|---|
The same shop with 1,000 visitors. Each purchase brings $100 of profit; the coupon gives $20 off to everyone who uses it. Profit is measured against a day without coupons.
In statistics the uplift is called the conditional average treatment effect (CATE): the average difference between "with a coupon" and "without" among people with the same features. The simplest way to estimate it is with two models: one learns the chance of buying on people who got a coupon, the other on the control group, and the forecasts are subtracted. That is how the uplift bot counted in the game.
When the discount costs money, the criterion changes again: a coupon pays if $100 × uplift is more than $20 × the chance of buying with a coupon. A coupon is worth +$30 per persuadable and −$18 per sure thing: they buy, but at a discount. So coupons for all 1,000 visitors add 43.5 purchases and at the same time lose $60. Level 3 of the game is built on this.
The question "did the coupon help?" really compares two worlds: one where the person got the coupon and one where they didn't. Nobody sees the second world; it can only be rebuilt on average from similar people who, at random, got no coupon. A control group is that borrowed second world.
Without it, it is easy to credit the coupon with purchases that would have happened anyway: "250 people got a coupon, 127 of them bought, so the coupon works". In fact without the coupon 150 of those 250 would have bought. The same trap lies in judging ads, medicines or training by those who got them without looking at those who didn't. How to set up a fair comparison is on the A/B test page.
Discount pop-ups are the everyday version of the puzzle: it is easiest to show them to people who already have something in the cart, and those are often the people who would buy without a discount.
The best-known sleeping dogs are in customer retention. A mobile operator calls the customers at highest risk of leaving, and the call reminds them that their contract has ended and they are free to go. Radcliffe and Surry (2011) describe a retention campaign after which churn rose from 9% to 10%. An uplift model found the 30% of customers on whom the call really worked; a campaign aimed only at them cut churn to under 8% and cost 70% less.
The same goes for election campaigns. Barack Obama's 2012 campaign, according to the journalist Sasha Issenberg (MIT Technology Review), gave voters a "persuasion score" from 0 to 10: how likely a conversation with a volunteer was to pull the person towards the candidate. The organisers warned outright that a conversation could also hurt the candidate.
US Bank mailed its customers paper letters offering a high-value product. Mailing everyone was profitable. Mailing the 30% chosen by a response model gave almost exactly zero extra sales: the model found people who would have bought anyway. An uplift model picked a different 30%, and they produced 90% of the extra sales of mailing everyone.
In his paper Paul Holland writes "Fundamental Problem of Causal Inference" in capitals and sets it out as a statement of its own: you cannot observe both outcomes on the same unit, and therefore you cannot observe the effect itself.
Radcliffe and Surry's first paper on modelling the uplift was presented not at a marketing conference but at "Credit Scoring and Credit Control VI" at the University of Edinburgh. Back then the method was called "differential response analysis".
Victor Lo called his approach the "true lift model". Since then the same idea has gone by uplift, net lift, incremental and differential response modelling, and in politics persuasion modelling.
Radcliffe described the Qini measure in 2007 as a generalisation of the Gini coefficient: the Gini compares a response model with random choice through the area under a curve, and the Qini does the same for the uplift curve.
In March 2008 the analyst Kevin Hillstrom published data from an email test on 64,000 customers: a random third got an email about men's merchandise, a third about women's, and a third got nothing. The data are still the standard teaching example for uplift modelling.
Share your own explanation, an objection or a variant of the puzzle. You need a GitHub account to comment.
Is “School maths” the right level for this puzzle?
Thank you! This helps set the levels right.
I keep adding new puzzles, new ways to explain them and write-ups of the discussions they start. Leave your email and I'll write when something new is out. No spam, one-click unsubscribe.
The sign-up form will appear here soon.