Math paradoxes

Courses / A/B testing paradoxes / Check your test

Course · tool

Check your test

Enter the numbers from your dashboard and answer a few questions about how the test ran. The page runs the same checks the course teaches: the traffic split, the result, the size of the test and the process. At the end comes a verdict in plain words, with its reasons. The "Share" link keeps your numbers, so a colleague sees the same report.

Step 1Dashboard numbers
Step 2The test plan
Step 3How the test ran
Was the result taken on the planned day, not when the dashboard hit 95%?
Did the test run in whole weeks?
Did B's share stay constant for the whole test?
Is conversion counted over all visitors, not at a later funnel step?
More than two variants in the test?

The check report

Verdict

    Put it into a report for your boss →

    What this check cannot see

    The page trusts your numbers and answers. Three things it cannot check, and each of them can flip the verdict:

    How it is computed

    Traffic split: a chi-square test with one degree of freedom against the planned share; an alarm if the is below 0.001, as in chapter 13.

    Result: a two-sided z-test for two proportions with a pooled rate; confidence = 1 − p. A 95% for the difference in conversion, divided by A's conversion. With k slices and v new variants, each comparison's threshold is 5% divided by k·v, as in chapters 3 and 4.

    Chance to beat: a flat Beta(1, 1) starting assumption for both rates and numerical integration over a 4,000-point grid; the expected loss is the of max(pA − pB, 0), in conversion points, as in chapter 11.

    Size: the sample formula of chapter 9 (two-sided 5%, 80% power, A's conversion as the base), with the planned share for an uneven split. Power is the share of tests of this size that would notice a lift of exactly the MDE.