Experiment design and power

The idea

Most "failed" A/B tests were lost before launch: they never had enough traffic to see the effect they were hunting. Power analysis is the pre-flight check that the experiment, as designed, can actually answer its question.

The vocabulary, in one frame: choose $\alpha$ (false positive tolerance, usually 0.05) and target power $1 - \beta$ (probability of detecting a real effect, usually 80%). Then for a true difference $\delta$ between arms:

Setting the detection requirement gives the two master formulas. Sample size per arm to detect $\delta$:

$n = \frac{(z_{1-\alpha/2} + z_{1-\beta})^2 \cdot 2\sigma^2}{\delta^2}$

With the standard conventions, $(1.96 + 0.84)^2 \approx 7.85 \approx 8$, so:

$n \approx \frac{16\,\sigma^2}{\delta^2} \quad \text{per arm (the rule of sixteen)}$

Inverting gives the minimum detectable effect at a fixed $n$:

For a conversion metric, $\sigma^2=p(1-p)$.

The economics live in the exponents: $n$ scales linearly in variance but with the inverse square of the effect size. Halving the MDE quadruples the required traffic — which is why "let's also detect tiny effects" quietly turns a two-week test into a two-quarter one, and why variance reduction techniques (CUPED, binary instead of continuous metrics, winsorizing whales) are worth real engineering effort: cutting $\sigma^2$ by 30% cuts traffic requirements by 30%.

MDE is a business input, not a statistical output: it is the smallest lift that would justify shipping. Deciding it after seeing results is how teams talk themselves into noise.

Worked example

Checkout redesign. Baseline conversion $p_1 = 0.10$; the smallest lift worth the engineering cost is one absolute point, $p_2 = 0.11$. Standard $\alpha = 0.05$ (two-sided), 80% power.

Sanity check with the rule of sixteen using $\bar{p} = 0.105$: $16 \times 0.105 \times 0.895 / 0.0001 \approx 15{,}000$ — same answer to within rounding.

Convert to calendar time: 10,000 eligible users/day, split 50/50 → 5,000 per arm per day → sample complete in ~3 days. Run two full weeks anyway: weekday/weekend mix, weekly seasonality, novelty decay. Sample size is a floor, not a stopping rule.

Now the negotiation table — required $n$ per arm at this baseline:

That last row is the conversation that matters: for a 0.2-point MDE you do not just "run longer" — a quarter-long test blocks the roadmap and drifts with seasonality. You change the design: a more sensitive proxy metric, CUPED, or accept a larger MDE.

Cheat sheet

• Sample size per arm (means): $n = (z_{1-\alpha/2} + z_{1-\beta})^2 \, 2\sigma^2 / \delta^2$; rule of sixteen: $n \approx 16\sigma^2/\delta^2$ (α = 0.05, 80% power).
• Proportions: replace $2\sigma^2$ with $p_1(1-p_1) + p_2(1-p_2)$; quick form $16\,\bar{p}(1-\bar{p})/\delta^2$.
• MDE at fixed $n$: $(z_{1-\alpha/2} + z_{1-\beta})\sqrt{2\sigma^2/n} \approx 2.8\,\sigma\sqrt{2/n}$.
• z-values: $z_{0.975} = 1.96$, $z_{0.80} = 0.84$, $z_{0.90} = 1.28$ (90% power → multiplier $(1.96+1.28)^2 \approx 10.5$, i.e. $n \approx 21\sigma^2/\delta^2$).
• Scaling laws: $n \propto \sigma^2$ (linear) and $n \propto 1/\delta^2$ (quadratic) — halve the MDE, pay 4x the traffic.

Design checklist: randomization unit = analysis unit · MDE agreed with product pre-launch · duration ≥ max(sample-size days, 1-2 full weeks) · guardrail metrics chosen · stopping rule fixed (no peeking without a sequential design).

Gotchas:

• Post-hoc power at the observed effect is a restatement of the p-value — report the CI instead.
• $\sigma$ for revenue-type metrics is huge → conversion/binary metrics are often 5-10x cheaper in traffic for the same decision.
• Reaching $n$ in 3 days does not license stopping in 3 days — weekly cycles and novelty effects bias short tests.
• Multiple variants split traffic: $k$ arms need roughly $k \times n$ users total, plus a multiple-testing correction that raises $n$ further.