Does the feature work? Impact without an experiment

"Feature users retain twice as well. So the feature doubles retention?"

Nobody hands a feature out at random. Users find it and turn it on themselves, and those who already use the product a lot do so more often. So a comparison of "with the feature against the rest" compares people, not the feature.

Below are 10,000 users of three kinds: rare, occasional and frequent visitors. The button does nothing. The slider sets how much more often active users find it.

• A comparison of "with the feature against without" is a correlation. Before crediting the gap to the feature, ask how these people differed before it.
• The simplest check is to look at the same people before the launch: if the feature's users retained better back then too, the gap is not from it.
• The same trap outside the product, with app customers and an install bonus, is in chapter 9 of the course "How not to fool yourself with data".

Deeper: how selecting people creates a link where there is none.

• Correlation

  A number from −1 to +1 showing how closely two quantities move together.

• Sampling bias

  A sample some people get into more easily than others, so it describes everyone badly.

"41% of users tried the feature. Is that a success?"

The share of people who tried a feature is counted first: it grows fast and looks good on a slide. But one can try it out of curiosity, because of a pop-up, or by accident. Value shows only in who comes back to the feature.

• The share who tried measures how the feature was shown, not what it is. Compare it only across the same way of showing.
• Look further down the funnel: how many came back in week two and how many use it a month later. Whether the feature changed the product's retention as a whole, these numbers don't tell either: that is part 2.

"DAU/MAU went from 20% to 30%. Do people get more from the product?"

The economist Charles Goodhart noticed that a figure people start to push on stops showing what it used to show. The anthropologist Marilyn Strathern put it shorter: when a measure becomes a target, it ceases to be a good measure. In a product the easiest things to move are counts of visits and minutes: DAU, sessions, time in the app.

• Before the launch, pick one primary metric that counts value for a person: a deal, a purchase, a task done per user. Keep the share who tried and the minutes as secondary.
• Add two or three guardrails that must not get worse: uninstalls, push opt-outs, complaints, speed. A win in the primary metric with a failed guardrail is not a win.

"We launched the feature on 1 September, and September's conversion is 12% above August's. There's the effect."

A "before and after" comparison is simple and therefore the most common. But in the same month many things change: the season, advertising, prices, other teams' releases. All of it lands in the feature's "effect".

• Compare with the same period last year, not with the previous month: that shows the season.
• Keep a shared calendar of changes: every team's releases, campaigns, prices, outages. Before crediting a metric's move to the feature, check what else happened that week.
• Best of all is a group that lived without the feature in the same month: then the season and the campaign act on both groups alike (chapters 6–7).

This builds on what you haven't done yet: chapter 4, “Before and after launch”

"Retention fell to a one-year low, we rushed out a fix, and it recovered. The fix worked!"

Weekly metrics wander even with no changes at all: different people, weather, holidays. And fixes and new features are often launched exactly when the metric is at its worst. That is the trap of this chapter.

Below is the retention of a product where nothing changes: about 40% every week plus noise. The team ships a "fix" every time a week is the worst of the last half year. The fix does nothing.

• Compare with the normal level before the drop, not with the worst week.
• If the fix is real, it should hold the metric above normal for weeks, not just bring it back. Better still, leave a small group without the fix (chapter 10).

Deeper: the same force as a catalogue puzzle. The course "How not to fool yourself with data" has a general chapter on it too.

This builds on what you haven't done yet: chapter 4, “Before and after launch”

"Android got the feature in March, iOS only in May. Is that bad for the analysis?"

On the contrary, it is a gift. While iOS waits, it lives in the same season and under the same campaigns, but without the feature. And when the feature reaches it too, the effect can be seen a second time.

• If the feature rolls out gradually anyway, plan the waves as an experiment: write down the order of countries or platforms and the dates in advance, and count the metric for whole waves, not only for the feature's users.
• Better to draw the order of the waves by lot. If the strongest country goes first, it may have been growing faster anyway.
• A gradual rollout inside an A/B test has its own trap when B's share changes from day to day: see chapter 5 of the course "A/B testing paradoxes".

This builds on what you haven't done yet: chapter 6, “A staggered rollout”

"In the country with the feature retention grew by 6 pp. But it also grew where there is no feature. So what is the effect?"

Chapter 6 already computed a difference in differences; now let us take it apart. You take the change in the group with the feature and subtract the change in the group without it over the same time. The second change is the season, campaigns and everything else that acted on both groups.

• Before computing, look at several months before the launch. If the lines run in parallel, the comparison deserves more trust; if they drift apart, look for another control group.
• Pick a similar control: a country with a similar product and audience, the same platform, the same plans. Several control groups are better than one.

This builds on what you haven't done yet: chapter 1, “The feature's users”

"Gold customers spend twice as much. Does gold status pay off?"

When a feature or a status starts at a threshold of points, orders or spend, a comparison of "with it against without" again compares different people, as in chapter 1. But right at the threshold a natural experiment appears: a customer with 995 points and one with 1,005 are almost the same, yet only the second has the status.

• A threshold works only if customers can't place themselves precisely around it. If points can be bought to jump over it, the keenest end up above it, and the comparison is unfair again. A check: the number of customers near the threshold should be about the same on both sides.
• The effect at the threshold applies to customers near the threshold. For those with 300 points, the status may act differently.

This builds on what you haven't done yet: chapter 1, “The feature's users”

"We matched each feature user with a non-user: the same plan, the same account age. Is the comparison fair now?"

When there is no group without the feature, the most common move is to take similar people. Matching on plan, country and account age removes the part of chapter 1's self-selection that the data can see. The part it can't see — motivation, need, habits — stays.

• A matched comparison is a hypothesis with an upper bound, not an effect. If the gap vanished after matching, that is good news; if it remains, it may still be self-selection.
• The best variable to match on is the metric itself before the launch: the user's retention or activity in the month before the feature. It carries part of the invisible motivation.

"We show the feature to everyone, but want to know its effect over three months. How many people should we leave without it?"

A holdout is the simplest honest way to measure a feature you have already decided to ship: a small random share of users stays without it for a few months. The smaller it is, the cheaper, but the noisier the comparison.

• The holdout is drawn by lot before the launch, not from those who "didn't update yet": they differ on their own, as in chapter 1.
• Sizing is the same decision as sizing an A/B test: first the effect worth seeing, then the number of people. The calculator is in chapter 9 of the course "A/B testing paradoxes", and a holdout for a whole year of changes is in chapter 17.

Deeper: why the noise between identical groups is so large.

• Standard error

  How far an average or a rate computed from a sample usually misses the truth: σ divided by the square root of the sample size.

• Confidence interval

  A range around a measured number, built by a method that covers the true value in a chosen share of samples, such as 95%.

• Statistical power

  The probability that a test notices a real effect of a given size. A power of 80% means the test misses one working effect in five.

"Over the year, the retention of active users grew from 60% to 80%. Has the product got better?"

New users leave most often: many take a look and never come back. Those who have stayed for half a year almost all stay on. So the retention of all active users together depends on how many newcomers are among them, not only on the product.

• Count retention by cohort: "of those who signed up in March, on day 30 there remained…". Compare cohorts of the same age before and after the launch.
• Judge a feature for long-standing users on cohorts that got it at the same age, not on all active users together.

Deeper: the same trap as the puzzle of the planes that came back. Survivors in reports and surveys are in chapter 6 of the course "How not to fool yourself with data".

This builds on what you haven't done yet: chapter 2, “Tried ≠ helped”

"'Quick reply' was used 50,000 times in a week. So it gave 50,000 replies?"

A new feature often does what an old one already did, only differently. Then part of its uses are actions that simply moved from the old way to the new. The feature's counter grows while the total number of actions barely moves.

• Measure the whole: all replies, all orders, all revenue, not the actions through the new button.
• If the new feature only speeds up an old action, its value lies in time or quality, not quantity. Then pick the metric to match (chapter 3).

"The new checkout raised conversion on both phones and computers. But total conversion fell. Is the feature harmful?"

Total conversion is the average of the groups, weighted by their shares. If after the launch the number of people in each group changed, the total may move the other way from every group on its own. This is Simpson's paradox.

• When the total and the groups disagree, first check whether the mix changed: devices, traffic sources, new and old users, countries.
• Compare groups with groups, and recompute the total with the old shares. But a slice chosen after seeing the data is only a hypothesis (chapter 3 of the course "A/B testing paradoxes").

Deeper: Simpson's paradox as a catalogue puzzle, and in sales reports, chapter 4 of the course "How not to fool yourself with data".

This builds on what you haven't done yet: chapter 7, “Difference in differences”chapter 9, “Matched groups”chapter 10, “A long holdout”

"There's no time for an A/B test. We'll launch and look at the numbers later."

The methods of part 2 are useful, but each rests on an assumption: parallel lines, an honest threshold, similar groups. An experiment needs none of them. Not every change is worth a test, but some are worth testing even when it is inconvenient.

• Insist on an experiment when: a specific effect in money is promised; a budget, hiring or the roadmap depends on it; the feature is supposed to pay for itself; or the methods of part 2 give different answers.
• How to plan a test and read its result is in the course "A/B testing paradoxes", and you can check your own numbers on the page "Check your test".

This builds on what you haven't done yet: chapter 1, “The feature's users”chapter 3, “A metric easy to move”chapter 4, “Before and after launch”chapter 5, “After the worst week”chapter 6, “A staggered rollout”chapter 12, “Cannibalisation”

"Here are six launch reports. Which of them should we believe?"

Each report holds exactly one trap from the course, and one holds none at all. Pick what is wrong with it. After you answer, an explanation appears with a link to the chapter that covered it.

The same questions as a questionnaire that gives the risks and an estimation method for your launch are on the page "Can you trust this impact?". Handy to open before every report and to send to colleagues.