Hypothesis testing and p-values

The idea

Hypothesis testing is proof by statistical contradiction. Assume the boring world — the null hypothesis $H_0$ (no effect, no difference) — and ask how surprising your data would be in that world. Surprising enough, and you reject the boring world in favor of the alternative $H_1$.

The measure of surprise is the p-value:

$p = P(\text{test statistic at least as extreme as observed} \mid H_0)$

Every word carries weight. Given $H_0$: the calculation lives entirely in the null world; it says nothing directly about whether $H_0$ is true. At least as extreme: it is a tail probability, not the probability of your exact data. The standard recipe:

1. State $H_0$ and $H_1$, and pick $\alpha$ (the false positive rate you will tolerate — 0.05 by convention, not by law) before looking.

2. Compute a test statistic whose distribution under $H_0$ is known — usually via the CLT. For two proportions:

   $$z = \frac{\hat{p}_B - \hat{p}_A} {\sqrt{\hat{p}(1-\hat{p})(1/n_A + 1/n_B)}}.$$

3. Convert to a p-value; reject $H_0$ if $p < \alpha$.

The error vocabulary: Type I ($\alpha$) — reject a true null; Type II ($\beta$) — miss a real effect; power $= 1 - \beta$ — the probability of catching a real effect of a given size.

The traps — this is what interviews actually test:

• $p$ is not $P(H_0 \mid \text{data})$. Reversing the conditional needs a prior (Bayes). In a world where most tested ideas are duds, most $p < 0.05$ results can still be false positives.
• $p > 0.05$ is not evidence of no effect — it may just be low power. Report the confidence interval; "CI spans zero and everything we care about" means "inconclusive", not "null confirmed".
• $1 - p$ is not the probability the effect is real.
• Significance is not importance: with $n$ in the millions, trivial effects become significant; with $n$ in the dozens, important effects do not.
• The guarantee holds only for the procedure as pre-registered: one look, one metric, one comparison. Peeking, metric shopping, and subgroup mining silently multiply $\alpha$.

Worked example

A/B test on conversion. Control: 500/5,000 = 10%. Treatment: 550/5,000 = 11%.

Pooled rate: $\hat{p} = \frac{500 + 550}{10000} = 0.105$.

$\text{SE}_0 = \sqrt{0.105 \times 0.895 \times \left(\tfrac{1}{5000} + \tfrac{1}{5000}\right)} \approx 0.00613$

$z = \frac{0.11 - 0.10}{0.00613} \approx 1.63 \quad\Rightarrow\quad p \approx 2\,(1 - \Phi(1.63)) \approx 0.10$

Two-sided $p \approx 0.10 > 0.05$: fail to reject. The right conclusion is not "the treatment does nothing" — the 95% CI for the lift is roughly $(-0.2, +2.2)$ percentage points, which contains both zero and a lift that would be a clear business win. The experiment was underpowered for a 1-point effect (it had roughly 37% power); it needed about 15,000 users per arm to detect that lift reliably at 80% power. The honest verdict: inconclusive, and the fix is sample size, not narrative.

Cheat sheet

• p-value: $P(\text{data at least this extreme} \mid H_0)$. Small $p$ → data implausible under $H_0$.
• Two-proportion z: pooled $\hat{p} = \frac{x_A + x_B}{n_A + n_B}$, $\; z = \frac{\hat{p}_B - \hat{p}_A}{\sqrt{\hat{p}(1-\hat{p})(1/n_A + 1/n_B)}}$.
• One-sample mean: $t = \frac{\bar{x} - \mu_0}{s/\sqrt{n}}$ with $n - 1$ df; use $t$ when $n$ is small, $z$ is fine for large $n$.
• Critical values (two-sided): $z = 1.645$ (10%), $1.96$ (5%), $2.576$ (1%).

Trap table:

• False discovery intuition: 1,000 tests, 10% real effects, power 0.8 → true positives $= 100 \times 0.8 = 80$; false positives $= 900 \times 0.05 = 45$; so 36% of "wins" are false even with everything done correctly.
• Fix $\alpha$, the metric, the test, and the stopping rule before launch; everything after that is the domain of multiple-testing corrections.