Can you trust this impact?

What this questionnaire cannot see

• Noise. Even a fair comparison can show a gap by chance if there are few people. How many people you need to see an effect is in chapter 10; noise between identical groups is in chapter 1 of the course "A/B testing paradoxes".
• Whether the split is really random. If the holdout was made of those who "didn't update yet", it differs on its own. For an A/B test, check the split on the page "Check your test".
• That the metric was chosen before the launch. If the primary metric was picked from twenty after the launch, one of them "wins" by chance (chapter 3 of the course "A/B testing paradoxes").