A/B testing pitfalls

The idea

A p-value is a contract with fine print: one pre-registered look, at one pre-registered metric, on a healthy randomization, measuring a stable effect. Every classic A/B pitfall is a violation of one clause. The math is rarely wrong — the experiment stopped satisfying the math's assumptions.

1. Peeking (optional stopping). Test statistics wander. Check daily and stop at the first $p < 0.05$, and you give noise many chances to cross the line: the realized false positive rate climbs from 5% to 20-30% over a few weeks of daily looks, and approaches certainty as looks accumulate. Fixes: commit to a horizon and evaluate once, or adopt a design built for continuous monitoring (group-sequential with alpha spending, mSPRT / always-valid p-values). Monitoring guardrails daily is fine — deciding success daily is not.

2. Multiple testing. Twenty metrics, five segments, three variants — every comparison is another lottery ticket for a false positive. At $m$ independent tests, $P(\text{at least one false positive}) = 1 - 0.95^m$: 64% at $m = 20$. Fixes: one pre-registered primary metric; Bonferroni ($\alpha/m$) when any single false claim is costly; Benjamini-Hochberg for dashboards where controlling the false discovery rate is enough; treat subgroup wins as hypotheses for the next test, not findings from this one.

3. Sample ratio mismatch (SRM). You designed 50/50 but got 50.6/49.4 on 100,000 users. That is not rounding error — a chi-square test puts it around $p \approx 10^{-4}$. Something upstream is deleting users asymmetrically (redirect latency, bot filters, crashes, caching bugs), and whatever deletes them is almost never independent of the metric. An SRM means the comparison is corrupted; the only correct move is to find the mechanism and rerun. Platforms auto-check this because humans rationalize it away.

4. Novelty and primacy effects. Early treatment effects are often transient: users poke at anything new (novelty inflates early lift) or stumble over change (primacy inflates early harm). Both decay. A two-day readout measures the reaction to change, not the value of the feature. Fix: cover the important calendar cycles for your population (often full weeks), and plot effect by days-since-first-exposure. A real effect can ramp, decay, or vary with learning; novelty is one possible explanation for a short-lived early response, not the only permitted shape.

Overarching all four: the winner's curse. Conditioning on significance selects lucky noise, so significant effects — especially from underpowered tests — are overestimates. Expect shipped features to underdeliver their readouts unless you power properly and validate with holdbacks.

Worked example

SRM check. Design 50/50; observed 50,600 vs 49,400 ($N = 100{,}000$, expected 50,000 each):

$\chi^2 = \frac{(50600 - 50000)^2}{50000} + \frac{(49400 - 50000)^2}{50000} = \frac{360000}{50000} \times 2 = 14.4$

With 1 degree of freedom, $p \approx 1.5 \times 10^{-4}$. A healthy randomizer produces a split this lopsided about once in 7,000 experiments — this experiment is broken, whatever its p-value says.

Multiple testing. The same test tracked 20 metrics. Expected false positives at $\alpha = 0.05$: $20 \times 0.05 = 1$. Probability of at least one: $1 - 0.95^{20} = 0.64$. Under Binomial(20, 0.05), the modal count is actually one false positive ($P(0)=.36$, $P(1)=.38$, $P(2)=.19$), so "a metric moved significantly!" is unsurprising — it is close to the most likely outcome of a do-nothing treatment, not a fluke. Bonferroni would demand $p < 0.0025$ per metric; BH would rank the 20 p-values and admit those with $p_{(i)} \leq 0.05\, i/20$.

Peeking intuition. A fixed-horizon test has one draw from the null distribution of $z$. Daily peeks for 14 days draw a correlated path and ask whether it ever exceeded 1.96 — and a random walk crosses any fixed boundary far more often than a single endpoint does. That "ever crossed" event is why sequential designs widen early boundaries (e.g. O'Brien-Fleming spends almost no $\alpha$ in week one).

Cheat sheet

Key numbers:

• $P(\text{at least one false positive})$ across $m$ tests: $1 - 0.95^m$ → 23% ($m=5$), 40% ($m=10$), 64% ($m=20$).
• SRM chi-square (2 arms): $\chi^2 = \sum (O_i - E_i)^2 / E_i$, 1 df; worry below $p \approx 0.001$.
• Daily peeking for 2 weeks ≈ 3-5x the nominal false positive rate.

Pre-launch contract (writes itself into the run-book): primary metric · $\alpha$, power, MDE · horizon date · guardrails + circuit-breaker thresholds (the only legitimate early stops) · segments declared exploratory. If a decision rule is not in the contract, it is not a decision rule — it is a rationalization waiting to happen.