A report like this arrives every quarter: four lines and four decisions. Every number in it is true, and every one can lead you astray. The course teaches you to ask a report a few simple questions. The first part is about reading the numbers themselves: averages, rankings, comparing groups. The second is about who got into the data and who is missing. The third is about conclusions: is it a cause or a coincidence, who gets credit for a sale, how many alarms are real, and are the numbers made up. At the end comes an exam of six reports and a verdict on this one.
≈ 1–1.5 hours, in 10-minute parts if you like; your progress is saved.
This course is about data that is simply collected: reports, surveys, cohorts. Experiments have a course of their own, "A/B testing paradoxes".
Done 0 of 13 chaptersCatalogue puzzles solved: 0 of 9
The sum of all values divided by how many there are. One very large value shifts it noticeably (chapter 1).
Median
The middle value when all are lined up from smallest to largest: half are below, half above.
Conversion rate
The share of visitors who bought: 50 purchases per 1,000 visitors is 5%.
Sample
Those who were measured or asked, as opposed to everyone we want to learn about.
Response rate
How many of those invited actually answered the survey (chapter 5).
Cohort
Customers who arrived in the same period, say in January. A cohort shows who stayed and who left (chapter 6).
Correlation
Two numbers move together. That does not yet mean one causes the other (chapters 7 and 9).
Attribution
The rule a report uses to hand sales out to ad channels, for example "last click" (chapter 10).
ROAS
Revenue per unit of ad spend: ₴1.3M of revenue for ₴108K spent is a ROAS of 12.
False alarm
The filter fired, but nothing happened: an honest transaction was blocked (chapter 11).
Part 1 · no formulas
How to read a report
Four situations where the number in a report is true and the conclusion drawn from it is wrong. Each chapter starts with a question: guess first, then check it on a live picture. There are no formulas here.
1
The average order is up 24%
"The average order is up 24%. Are customers buying more?"
The mean is the most popular number in reports: the sum of all orders divided by how many there are. But it is very sensitive to one large order. The median is the order in the middle when all orders are lined up from smallest to largest: half are smaller, half larger.
In Q2 there were 1,000 orders with an average of ₴2,000. In Q3 there were 1,000 orders again, one of them a corporate order for ₴482,000. The average order became ₴2,480. What happened to the ordinary customer's order?
One order of ₴482,000 is ₴480,000 more than an ordinary order. Divided by 1,000 orders, that is +₴480 on the average, which is the whole increase. The other 999 orders still average ₴2,000. Move the slider below.The whole increase comes from one order: ₴482,000 is ₴480,000 more than an ordinary order, and divided by 1,000 orders that is +₴480 on the average. The other 999 orders still average ₴2,000. Move the slider below.
The 1,000 orders of Q3
meanmedian
Average order–
Median–
Mean without the largest–
Orders are almost never spread symmetrically: many small ones and a few large ones. So the average order is usually above the median even without corporate clients. Nearly all money and time behaves this way: revenue per customer, salaries, call durations, delivery times.
The ordinary customer's order did not grow: the median barely moved, and the mean without the largest order stayed at ₴2,000. The +24% is one deal. It deserves its own line, as a deal, not as a trend.
Ask for the median next to the mean. If they differ a lot, the data has a few very large values.
Ask what the mean is without the largest customers, for example without the top 1%.
Show big deals on a separate line: that way you see both the trend and the deal.
StoryIn a 1954 book, Darrell Huff describes a real-estate agent. To a buyer he says the neighbourhood's average income is $15,000: a mean pushed up by a few millionaires with weekend homes. A year later, arguing that the neighbours cannot afford a tax rise, he gives the same area an "average" income of $3,500. Now it is the median. Both numbers are true.Source: Darrell Huff, "How to Lie with Statistics" (1954), chapter 2, "The Well-Chosen Average".
"Here are the top 10 stores by conversion. Shall we roll their practices out across the chain?"
Rankings are a favourite genre of reports: the best branches, the worst managers, the top regions. But in a ranking by a rate (conversion, return rate, churn) small units have a quirk that is often forgotten.
A chain has 100 stores, all equally good: each one truly converts at 5%. Their traffic differs, from 200 to 20,000 visitors a month. Who will end up in the month's top 10 by conversion?
Half right. Small stores do take the top, but the bottom too: their conversion jumps both ways. Large stores stay near 5% and are rarely at either end. Check below.A store with 200 visitors expects 10 purchases, and a few extra or missing ones push its conversion to 3% or 7%. A large store has 1,000 purchases and is almost always near 5%. So both ends of the ranking go to small stores. Check below.Small stores are both at the top and at the bottom. A store with 200 visitors expects only 10 purchases, and a few extra or missing ones push its conversion to 3% or 7%. A large store with 1,000 purchases is almost always near 5%. Check below.
100 identical stores, one month
top 10bottom 10the rest
Small stores in the chain–under 1,000 visitors
Small stores in the top 10–
Small stores in the bottom 10–
The smaller the group, the more its rate wobbles around the true one. This is the law of large numbers, and the typical size of the wobble is called the standard error. Four times fewer people make the wobble twice as large.
Both the top 10 and the bottom 10 are mostly the smallest stores, though all the stores are identical: usually 7–8 out of 10, while only a third of the chain is small. Their place was set by the number of visitors, not by the quality of their work. Next month they would swap places.
Before learning from the leaders of a ranking, look at the size of each. If both the top and the bottom are small, the ranking shows noise.
Show the count next to the rate, and judge small units over a longer period or together.
StoryIn the late 1990s, people in the US noticed that many of the best schools in the rankings were small, and the Bill & Melinda Gates Foundation began funding the break-up of large schools into small ones. The statistician Howard Wainer looked at all 1,662 schools in Pennsylvania: among the 50 best on fifth-grade tests there were four times as many small schools as chance would give. But there were too many among the 50 worst as well. In 2005 the foundation moved away from the strategy.Source: Howard Wainer, "The Most Dangerous Equation", American Scientist 95(3), 2007.
"The best managers of Q1 got a bonus and sold less in Q2. Did the bonus make them lazy?"
A quarter's result has two parts: skill and luck. Skill carries over to the next quarter; luck does not. Whoever ends up at the very top usually had both. As in chapter 2, luck here is ordinary noise.
A department has 40 managers. In Q1 the best five hit 124% of plan on average, and the whole department 100%. There were no bonuses or changes in the work, and skill and luck matter about equally. What will these five show in Q2?
Not all the way to the average: the skill is real, and it stays. Only the part that luck gave rolls back, roughly half the lead. Check below.Skill carries into Q2, luck does not. The five reached the top mostly because they had both skill and luck, so in Q2 they come back halfway to the average on average: from 124% to 112%. This is regression to the mean. Check below.Skill stays, luck does not. The five reached the top mostly because they had both skill and luck, so in Q2 they come back halfway to the average on average: from 124% to 112%. This is regression to the mean. Check below.
40 managers, two quarters
top 5 in Q1bottom 5 in Q1the rest
Top 5: Q1–
The same five: Q2–
Bottom 5: Q1 → Q2–
The worst five "improve" just the same, with no effort at all. If they were sent to a training course right then, the course would look brilliant. Anything done to people who just showed an extreme result (a bonus, a reprimand, coaching, a new system) will look stronger or weaker than it is. The only way to see this is to compare with a similar group that nothing was done to.
In Q2 the top five fall halfway back to the average on average, and the bottom five rise just as much. The bonus has nothing to do with it: when luck is half the story, this happens with no bonus at all. The more luck, the stronger the pull back.
Judge the effect of a bonus, a training or a programme for the "worst" against a similar group without it: regression acts on both.
Pick the best over several quarters, not one: over several quarters luck partly evens out.
StoryDaniel Kahneman tells of an instructor of Israeli Air Force pilots. He was sure that praise does harm: a cadet praised for a superb manoeuvre flies worse the next time, while one who was shouted at does better. Kahneman explained that the instructor was seeing regression to the mean: after an exceptionally good attempt the next one is almost always closer to ordinary, whatever you say.Source: Daniel Kahneman, "Thinking, Fast and Slow" (2011), chapter 17.
"The partner channel converts at 4.8%, search ads at 8.5%. Shall we cut the partners?"
A channel's overall conversion is an average over different visitors. New visitors rarely buy, returning ones often do. If one channel brings mostly new visitors and the other mostly returning ones, the overall numbers compare not the channels but the mix of their traffic.
The partner channel converts at 4.8%, search ads at 8.5%, with 5,000 visitors each. Can the partner channel be better among both new and returning visitors?
Partners bring 80% new visitors, ads only 20%. In each group partners are better: 3% vs 2.5% among new visitors and 12% vs 10% among returning ones. But new visitors rarely buy, and they are most of the partner traffic. This is Simpson's paradox. Move the traffic mix below.It can, in both groups at once. Partners bring 80% new visitors, ads only 20%. Among new visitors partners are better (3% vs 2.5%), among returning ones too (12% vs 10%), but new visitors rarely buy, and they are most of the partner traffic. This is Simpson's paradox. Move the traffic mix below.
Two channels, 5,000 visitors each
The partner channel is better in both groups. Its overall conversion is lower only because it brings new people, and they buy less often. Cutting it means losing exactly the new customers.
When groups are compared by a single rate, ask whether their mix is the same: new and returning, regions, devices, large and small customers.
Compare within like groups, and then decide separately how much of each group you want.
StoryIn autumn 1973 the University of California, Berkeley received 12,763 applications to its graduate school. 44% of men and 35% of women were admitted, which looked like discrimination. But within departments women were not admitted less often; overall, slightly more often. Women more often applied to departments that were hard for anyone to get into, with long queues for each place.Source: P. J. Bickel, E. A. Hammel, J. W. O'Connell, "Sex Bias in Graduate Admissions: Data from Berkeley", Science 187, 1975.
Deeper: the classic Simpson's paradox in the catalogue.
What you can do nowYou can tell a trend from one big customer (1), you see noise in rankings of small units (2), you don't confuse regression to the mean with the effect of a bonus (3), and you compare channels within like groups (4). So far we assumed everyone is in the data. The second part is about who got into the data and who is missing.
Part 2 · a few numbers
Who got into the data
A report counts only those it sees: who answered, who stayed, who passed a filter, who came at the peak hour. Four chapters about the people missing from the data and how they change the conclusion. There are a few more numbers, but everything can be counted on your fingers.
5
92% satisfied
"92% of customers are satisfied. Can we report that?"
A survey hears only those who answer. If satisfied customers answer more readily than unhappy ones (or the other way round), the share among the answers drifts away from the share among all customers. Such a sample is biased, and more answers do not help.
An in-app survey: 92% of answers say "satisfied". 7.6% of customers answered, and satisfied customers answer five times as readily as unhappy ones. How many of all customers are satisfied?
In the answers there are 11.5 satisfied customers per unhappy one (92 : 8). But satisfied customers answer five times as readily, so among all customers there are only 2.3 per unhappy one: 70% against 30%. Check below, and try increasing the number of customers.About 70%. In the answers there are 11.5 satisfied customers per unhappy one (92 : 8). Satisfied customers answer five times as readily, so among all customers there are only 2.3 per unhappy one: 70% against 30%. Check below, and try increasing the number of customers.
A survey that doesn't hear everyone
Customers
one surveythe true share, 70%
Answered–
Satisfied among answers–
Satisfied among all70%
About 70% of all customers are satisfied, not 92%. And no number of answers will fix that: with 100,000 customers the survey just shows the same 92% more precisely.
Next to a survey result, ask how many people answered and whether the opinion itself could affect who answers.
Cross-check with the behaviour of all customers: repeat purchases, returns, support tickets do not depend on who felt like answering.
It is more reliable to ask a small random sample with reminders, so that most of them answer, than to collect thousands of answers from volunteers.
StoryIn 1936 the Literary Digest mailed out 10 million ballots, got about 2.4 million back and predicted that Alf Landon would beat Franklin Roosevelt. Roosevelt won 46 of the 48 states. George Gallup, polling about 50,000 people, named the winner correctly. A later analysis showed that the Digest's main trouble was who answered: Landon's supporters were keener to mail their ballots back.Source: Peverill Squire, "Why the 1936 Literary Digest Poll Failed", Public Opinion Quarterly 52(1), 1988.
Deeper: how many people you need to ask when the sample really is random.
"We asked customers who have been with us a year: only 12% complain about the price. Can we raise prices?"
A survey of current customers hears only those who stayed. Those who were unhappy with something have mostly left already and will say nothing. This is survivorship bias: the data keeps those who passed a filter, and the reason for the filter vanishes from the data.
A year ago 10,000 people subscribed. Each month 15% of those who find it expensive leave, and 3% of the rest. After a year, 12% of those who stayed complain about the price. How many of those who arrived a year ago found it expensive?
Of those who find it expensive, only 14% remain after 12 months (85% stay each month, and 0.85 to the 12th power is 0.14). Of the rest, 69% remain. So 40% at the start become 12% among the survivors. The price is the main reason people left, and the survey cannot see it. Check below.About 40%. Of those who find it expensive, only 14% remain after 12 months (0.85 to the 12th power), of the rest 69%. So 40% at the start become 12% among the survivors. The price is the main reason people left, and the survey cannot see it. Check below.
A cohort: 10,000 subscribers over a year
find it expensivethe restleft
Find it expensive, among those asked–
Find it expensive, among all who came40%
Left over the price–
The price is the main reason for churn: the 40% of newcomers who find it expensive shrink to 12% of those who stay within a year. Raising prices on the strength of this survey means listening only to those the price did not scare away.
Ask those who leave why: a short question at cancellation, a call a week later.
Look at cohorts from the start: how many of those who arrived in January stayed, and who exactly left.
The same goes for successes. "What our best customers have in common", without those who left, shows not the cause of success but only the survivors.
StoryInvestment funds that earn poorly close down or merge into others and vanish from the statistics. So the average return of the funds "on the market" looks better than all funds really did. Edwin Elton, Martin Gruber and Christopher Blake followed every large US fund that existed in 1976 through to 1993: counting only the survivors overstates fund performance by about 0.9 percentage points a year.Source: E. J. Elton, M. J. Gruber, C. R. Blake, "Survivor Bias and Mutual Fund Performance", Review of Financial Studies 9(4), 1996.
"Among our top sellers, the pricier the product, the higher its rating. Customers believe pricier is better. Shall we raise prices?"
A "top sellers" report is not all products but those that passed a filter: buyers take a product if it is worth its money. A cheap product can sell with an average rating, while a pricey one makes the top only if it is really good.
In the market, a product's price and its rating are unrelated. Products with good value for money make the top sellers. What will a report on the top show?
A pricey product passes the filter only if its rating is high, while a cheap one passes with an average rating too. So in the top, pricier products are better on average, though in the market there is no link. This is Berkson's paradox. Check below.It will show pricier products rated higher. A pricey product passes the filter only if its rating is high, while a cheap one passes with an average rating too. The filter itself creates the link. This is Berkson's paradox. Check below.
400 products in the market and the top sellers
in the top sellersthe rest of the market
Correlation in the market–
Correlation in the top–
Rating in the top: cheaper half → pricier half–
The same goes for hiring. If you hire candidates whose interview and test-task scores add up to a high total, among those hired the two scores will look opposed: whoever shone at the interview is weaker at the test. Among all candidates there may be no such link at all.
In the top sellers, pricier products have higher ratings, though in the market price and rating are unrelated. The filter itself created the link, so this report is no ground for raising prices in the hope that "pricier is better". The stricter the filter, the stronger the false link.
A link in a selected group (top sellers, hired candidates, customers who reached checkout) says nothing about the link among everyone. Ask what the group was selected on.
If the selection depends on both things you compare, a link is almost guaranteed to appear.
"The gym's average occupancy is 30%. Why do members complain it is crowded?"
There are two different averages here. One is taken over hours: how many people are in the gym in an average hour. The other is taken over people: how many people an average visitor sees around them. The second is almost always larger, because the busy hours have more people there to see it.
A gym for 100 people is open 16 hours a day. According to the turnstiles, an average hour has 30 people in the gym. Most come in the evening. How many people does an average visitor see around them?
An hour with 80 people in the gym is 80 visitors who see a crowd. An hour with 8 people is only 8 who see an empty gym. So the average over visitors is weighted by the crowd, and it comes out at about 50, not 30. Check below.About 50. An hour with 80 people is 80 visitors who see a crowd, and an hour with 8 people is only 8 who see an empty gym. The average over visitors is weighted by the crowd. Check below.
One day of a gym for 100 people
average over hoursaverage over visitors
People in an average hour–
An average visitor sees–
Visitors at peak hours–when the gym has 60 people or more
An average visitor sees about 50 people around, not 30. Both numbers are true: the first is about the gym, the second about the members. Complaints count the second. Moving people off the peak lowers the second number, while the first stays the same.
Before answering complaints with a report, ask: an average over what? Over hours, groups, webinars, or over people?
The same holds for the size of groups and teams: the average class on a course is smaller than the class an average student sits in. The average wait over hours is shorter than the average customer's wait, because the queue is long exactly when there are the most customers.
StoryThe sociologists Scott Feld and Bernard Grofman showed this with one university's timetable: the average class in the timetable turned out much smaller than the class an average student sits in. Big lecture halls are few classes but many students, and they make up the "student" average. The university in its brochure and the student in the hall honestly quote different numbers.Source: S. L. Feld, B. Grofman, "Variation in Class Size, the Class Size Paradox, and Some Consequences for Students", Research in Higher Education 6(3), 1977.
Deeper: two catalogue puzzles about the same weighting.
What you can do nowYou ask who answered the survey (5), who left before the measurement (6), what the group was selected on (7) and what the average is taken over (8). The third part is about conclusions: when a link means a cause, who gets credit for a sale, how many alarms are real, and whether you can trust the numbers themselves.
Part 3 · a little harder
What follows from what
The costliest mistakes happen not when a number is wrong but when a wrong conclusion is drawn from a right number. Here you will need shares of shares and a little conditional probability, but the calculators do the maths for you.
"Customers with the app spend 2.3 times more. Shall we give a ₴500 bonus to everyone who installs it?"
When two numbers move together, that is correlation. But it does not say who pushes whom. The app is installed more often by those who buy a lot anyway: it is handier for regular customers. Then both the app and the big spending have one cause, loyalty. It is the same group mix as in chapter 4, only now it hides behind the words "thanks to".
Customers with the app spend ₴4,130 a quarter, without it ₴1,810. Regular customers install the app more often. How much will one new app user add per quarter?
The report can't tell that either. The app may help a little, and in our model it adds 5% to spending. The trouble is that the report does not separate this effect from who installs the app. Check below.The 2.3 times mixes two things: what the app itself gives and who installs it. In our model the app adds only 5% to spending, and the rest of the gap is regular customers, who install it more often. A new user will bring about ₴90 a quarter, not ₴2,300. Check below.The report can't tell. The 2.3 times mixes two things: what the app itself gives and who installs it. In our model the app adds only 5% to spending, and a new user will bring about ₴90 a quarter. Check below.
Spending per quarter, with the app and without
In the report: with the app / without–
A new user will add–per quarter
A ₴500 bonus pays back in–
The model is simple: 30% regular customers spend ₴5,000 a quarter, 70% occasional ones ₴1,200. 60% of the regulars installed the app, and 10% of the occasional ones. Even when the app gives nothing at all, the report shows a gap of 2.2 times.
A new app user will add about ₴90 a quarter. A ₴500 bonus pays back in about five and a half quarters, if at all: the 2.3 times is mostly a portrait of those who install the app, not what it gives.
How do you find the real effect? Offer the bonus to a random half of customers and compare the two halves. A random split makes the groups alike in their mix, so the difference between them is the effect of the bonus. This is an A/B test, and its own traps have a course of their own, "A/B testing paradoxes".
How to judge the impact of a new product feature when an experiment is impossible is the subject of a separate course for product teams, which is in the works.
When you hear "those who do X have more Y", ask: who usually does X? Is there one cause behind both?
Split into like groups. If the gap melts within the groups, it came from the mix, as in chapter 4.
Only an experiment with a random split checks a cause reliably.
StoryFor decades, observational studies showed that women taking hormone replacement therapy after menopause had less heart disease: in a large study of nurses the risk was 44% lower. The randomised Women's Health Initiative trial with 16,608 women found the opposite: 29% more coronary heart disease with the hormones, and the trial was stopped early. The women who chose the therapy were healthier to begin with.Source: M. J. Stampfer et al., New England Journal of Medicine 325, 1991; Writing Group for the WHI Investigators, JAMA 288(3), 2002.
"Retargeting brought 1,300 sales for ₴108,000, a ROAS of 12. Shall we double the budget?"
An attribution report hands sales out to channels. The most common rule is the last click: the sale goes to whatever the person clicked last before buying. Retargeting is shown to people who have already been on the site, that is, to those who are close to buying anyway. So it often becomes the last click before a purchase that would have happened without it.
In a month the site sold 2,100 orders. Last-click attribution credits retargeting with 1,300 of them. What happens to sales if you switch it off?
The last click shows who was next to the purchase, not who caused it. In our model 2,000 people would have bought without the ads, 1,200 of them clicked a retargeting ad on the way, and the ads truly added only 100 purchases. Switch them off, and sales fall by about 100. Check below.The report can't tell. The last click shows who was next to the purchase, not who caused it. In our model 2,000 people would have bought without the ads, 1,200 of them clicked a retargeting ad on the way, and the ads truly added only 100 purchases. Check below.
The month's 2,100 sales: who brought whom
would buy anyway, no clickwould buy anyway, but clicked an adbought because of the ad
Report: last-click ROAS–
In truth: revenue per ₴1 of ads–
Sales lost without the ads–
The average order in the model is ₴1,000, the budget ₴108,000. How do you find the real increase? With a control group: some people or regions are not shown the ads at all, and the difference in purchases between the groups shows how much the ads truly added. Ad platforms call this a lift test.
Without retargeting, sales fall by about 100, not 1,300. Each hryvnia of ads returns 93 kopecks of revenue, before the cost of the goods. Doubling the budget would only double the loss.
Attribution divides sales up, but it does not say how many would not have happened without the channel. For budget decisions, ask for the increase, not the attribution.
The channels that see people right before buying are the most overstated: retargeting, ads on your own brand name, coupons and cashback.
The increase is checked with a control group: people or regions who are not shown the ads.
StoryIn 2012 eBay switched off its ads for searches containing the word "ebay" on Yahoo! and Bing, keeping them on Google for comparison. The reports said these ads brought in plenty of sales. But 99.5% of the clicks they gave arrived anyway, through the free link right below the ad. A separate experiment with randomly chosen regions showed that eBay's other search ads also brought in less than they cost.Source: T. Blake, C. Nosko, S. Tadelis, "Consumer Heterogeneity and Paid Search Effectiveness: A Large-Scale Field Experiment", Econometrica 83(1), 2015.
11
The fraud filter fired
"The filter is 99% accurate and it blocked a transaction. So it's a fraudster?"
A filter's accuracy describes how it treats fraudsters and honest customers separately. But we want something else: how many of those it blocked are fraudsters. The answer also depends on how rare fraudsters are.
0.1% of transactions are fraud. The filter catches 99% of fraudsters and wrongly blocks 1% of honest transactions. What share of blocked transactions are really fraud?
Out of 100,000 transactions, 100 are fraud, and the filter catches 99 of them. But 1% of the 99,900 honest ones is another 999 alarms. That makes 1,098 blocked, and only 99 of them are fraud, that is 9%. Check below.Fewer than one in ten. Out of 100,000 transactions, 100 are fraud, and the filter catches 99 of them. But 1% of the 99,900 honest ones is another 999 alarms. Of the 1,098 blocked, only 99 are fraud, that is 9%. Check below.
100,000 transactions through the filter
fraudhonest customer
Alarms–
Real among alarms–
Honest customers blocked–
Each cell of the grid is 1% of the alarms. The share of real ones among the alarms is called a conditional probability: the probability of fraud given that the filter fired. Bayes' theorem calculates it, but picturing 100,000 transactions, as above, is enough.
Only 9% of blocked transactions are fraud: 91 out of every 100 alarms are honest customers. 99% accuracy does not save the day here: fraudsters are so rare that even 1% errors among honest customers outnumber everyone caught. A second independent check raises the share of real ones to 91%.
About any filter (anti-fraud, scoring, churn prediction, a spam detector) ask: how many alarms are real? It depends on how rare the event is.
An alarm needs a second, independent check, not an automatic block.
Count the cost of both kinds of error: a fraudster missed and an honest customer blocked.
StoryAt the 2017 Champions League final in Cardiff, South Wales Police scanned the crowd with facial recognition. The system produced 2,470 matches with its watch list, and 2,297 of them, more than 9 in 10, were false. Wanted people were very rare in a crowd of 170,000, so even rare errors outnumbered the true matches. The police stressed that no one was arrested because of a false match.Source: BBC News, "2,000 wrongly matched with possible criminals at Champions League", 5 May 2018.
"Can you tell from the amounts alone that an expense report is made up?"
In real data that spans several orders of magnitude (invoices, expenses, city populations), the first digit is spread unevenly: there are far more ones than nines. A person making numbers up usually does not think about this.
You take 10,000 real supplier invoices, from ₴50 to ₴500,000. What share of the amounts start with the digit 1?
Almost 30%: this is Benford's law. For an amount starting with 1 to start with 2, it has to double, while going from 9 to 10 takes only 11% more. So amounts that grow and multiply "live" on the one for a long time and rush past the nine. Check below.Almost 30%: this is Benford's law. For an amount starting with 1 to start with 2, it has to double, while going from 9 to 10 takes only 11% more. So amounts that grow and multiply "live" on the one for a long time. Check below.
The first digits of 500 amounts
these 500 amountsBenford's law
Start with 1–
Gap from Benford–on average per digit
Real invoices follow Benford almost exactly: about 30% start with 1 and only 5% with 9. Amounts made up at random are spread evenly and stand out clearly. But city taxi rides stand out too, though they are real: a range of ₴80 to ₴400 is too narrow.
A Benford check is a reason to look closer, not proof. Auditors start with it; they don't end with it.
It works only for data that spans several orders of magnitude and has no thresholds or caps. Prices, fares and amounts within a limit fail it, and that is normal.
Another sign of made-up amounts: too many just below a limit, for example ₴4,900–4,999 when everything from ₴5,000 needs approval.
StoryEconomists from Germany ran a Benford test on the deficit and debt data that EU countries sent to Eurostat. The data that departed most from Benford were Greece's, which in 2009 admitted it had understated its deficit. The digits proved nothing on their own, but they showed where to look first.Source: B. Rauch, M. Göttsche, G. Brähler, S. Engel, "Fact and Fiction in EU-Governmental Economic Data", German Economic Review 12(3), 2011.
What you can do nowYou can tell a link from a cause (9) and an increase from attribution (10), you can work out how many alarms are real (11), and you know how to spot made-up amounts (12). What remains is the exam: six reports, five of them with a trap from the course, and then the verdict on the report from the start of the course.
Finale
The exam and the verdict
Everything from the course is now in reports that could land on your desk.
"Here are six reports with proposals. What do we do with them?"
Five reports hold one trap from the course each, and one is clean. Choose what to do. After you answer, an explanation appears with a link to the chapter that covered it.
Report 1
B2B: revenue per clientQ2
Clients
120
118 the quarter before
Revenue per client
₴41,200
+31% on Q1
The team's conclusion: "clients have started buying more, let's raise the target". Among the new clients is the Megabud chain, whose contract is ₴1.4M.
Look at the median and at revenue without Megabud. Without it, the other 119 clients average ₴29,800, less than the quarter before (₴31,450). So ordinary clients are buying less, and the +31% is one deal (chapter 1).
Report 2
Branch support programmeQ2
The 10 worst branches of Q1, with a coach
+18%
The other 70 branches
+2%
Proposal: hire coaches for all 80 branches.
Not a conclusion: it may be regression to the mean. The branches were chosen precisely for being the worst in Q1, and the worst rise on their own the next quarter, on average (chapter 3). To judge the coach, compare with similarly weak branches without one, for example by giving coaches to a random half of them.
Report 3
Free delivery: a test4 weeks
Profit per customer, after delivery costs
+4%
interval +2% to +6%
Free delivery was shown to a randomly chosen half of customers: 25,012 against 24,988. The delivery cost is already subtracted. Proposal: switch it on for everyone.
You can act. There is no trap here: the groups were split at random, so their mix is the same (chapter 9), all customers are counted, not just those who answered or stayed (chapter 5), and the metric already includes the cost of delivery. Even the lower end of the interval is positive.
Report 4
Sales team: call conversionOctober
Olha
18.6%
800 cold and 200 warm calls
Petro
34.0%
200 cold and 800 warm calls
Proposal: hand some of Olha's clients over to Petro.
Compare within like groups. Olha is better both at cold calls (12% vs 10%) and at warm ones (45% vs 40%). Her overall conversion is lower only because she gets mostly cold calls (chapter 4).
Report 5
Newsletter and purchasesyear
Purchases per customer: newsletter subscribers
4.2
Purchases per customer: not subscribed
3.0
Customers subscribe of their own accord. Conclusion: "the newsletter adds 40% more purchases". Proposal: subscribe every customer automatically.
Check the cause with an experiment. Those who subscribe already like the shop, and they buy more often anyway (chapter 9). Send the newsletter to a random half of those not subscribed and compare the purchases of the two halves.
Report 6
Support: NPSyear
NPS
72
+15 in a year
Answered
900
This year the survey is sent after a closed ticket if the customer rated the conversation 4 or 5 stars. Conclusion: "customers have come to love our support".
Find out who is missing from the data. The survey sees only those who already gave 4 or 5 stars: unhappy customers never reach it. The 15-point rise may be a change in the selection rule, not in the support (chapter 5).
And now the report from the start of the course
Q3 reportfor the board
Average order
₴2,480
+24% on Q2
Satisfied customers
92%
an in-app survey
Customers with the app spend
×2.3
proposal: a ₴500 bonus for installing it
Retargeting
ROAS 12
proposal: double the budget
Verdict: all four numbers are true, and none of the four decisions should be made on them.
Average order: the +24% comes from one corporate order. Without it the mean stayed at ₴2,000, and the median barely moved (chapter 1). The deal deserves a line of its own.
92% satisfied: 7.6% answered, and satisfied customers answer five times as readily. Among all customers about 70% are satisfied (chapter 5). Ask a random sample, with reminders.
×2.3 with the app: regular customers are the ones who install it. A new user will add about ₴90 a quarter, and a ₴500 bonus won't pay back even within a year (chapter 9). Test the bonus on a random half of customers.
ROAS 12: this is last-click attribution. The real increase in the model is 100 sales, 93 kopecks of revenue per hryvnia of ads (chapter 10). Before doubling the budget, run a test with a control group.
✓
Before you decide on a report
Twelve questions, one from each chapter. Not every one applies to every report, but each will save you from a costly decision at least once.