Confidence intervals

The idea

A point estimate without an error bar is a guess. The confidence interval turns the standard error into a range of parameter values that are compatible with the data:

$\text{estimate} \; \pm \; z^* \times \text{SE}$

For a mean: $\bar{x} \pm 1.96\, s/\sqrt{n}$ at 95%. For a proportion: $\hat{p} \pm 1.96 \sqrt{\hat{p}(1-\hat{p})/n}$. The multiplier $z^*$ comes straight from the normal sampling distribution the CLT provides: 1.645 for 90%, 1.96 for 95%, 2.576 for 99% (swap in $t$ critical values when $n$ is small).

What "95%" means is the most-tested subtlety in statistics. The parameter is fixed; the interval is the random object. The guarantee is about the factory, not the widget: the procedure produces intervals that cover the truth in 95% of repeated samples. Once the numbers are in, this interval either covers or it does not — assigning it "95% probability" is, strictly, a Bayesian statement that requires a prior. (In practice, with reasonable priors the Bayesian credible interval is numerically similar, which is why the sloppy reading usually survives contact with reality.)

Two properties worth having at your fingertips:

• Test duality. When a confidence interval is constructed by inverting a particular family of two-sided tests, it contains exactly the null values that the matching tests fail to reject. Do not assume exact duality for an arbitrary pairing—for example, an unpooled Wald interval and a pooled two-proportion test can disagree near the boundary.
• Width economics. Width $= 2 z^* \sigma / \sqrt{n}$: quadruple the data to halve the interval; pay ~31% more width to go from 95% to 99% confidence.

CIs beat p-values as a reporting tool because they keep effect size in view. "Significant" could mean a microscopic effect measured precisely; "not significant" could mean an underpowered test whose interval still contains a fortune. The interval shows which.

Worked example

Same A/B numbers as the testing example — control 500/5,000 (10%), treatment 550/5,000 (11%):

$\text{SE}_{\text{diff}} = \sqrt{\frac{0.10 \times 0.90}{5000} + \frac{0.11 \times 0.89}{5000}} \approx 0.00613$

$\text{CI}_{95\%} = 0.01 \pm 1.96 \times 0.00613 = 0.01 \pm 0.0120 \;\approx\; (-0.002, \; 0.022)$

Reading it out loud: the data is consistent with anything from a 0.2-point loss to a 2.2-point gain. Three separate conclusions, all visible at a glance:

1. Contains zero → not significant at 5% (matches $p \approx 0.10$).
2. Contains +2 points (a 20% relative lift) → absolutely not "proof of no effect"; the test is underpowered for effects we care about.
3. Decision: keep collecting or re-size the experiment — the interval, unlike the p-value, says so directly.

To halve this interval's width you would need $4 \times 5000 = 20{,}000$ users per arm.

Cheat sheet

• General form: estimate $\pm z^* \times$ SE. Mean: $\bar{x} \pm z^* s/\sqrt{n}$. Proportion: $\hat{p} \pm z^* \sqrt{\hat{p}(1-\hat{p})/n}$. Difference: SE $= \sqrt{\text{SE}_1^2 + \text{SE}_2^2}$.
• Multipliers: 90% → 1.645 · 95% → 1.96 · 99% → 2.576. Small $n$: use $t_{n-1}$ critical values.
• Duality: value outside the $(1-\alpha)$ CI ⟺ rejected at level $\alpha$ (two-sided).
• Width: $\propto z^* / \sqrt{n}$ — halve it with $4\times$ data; 99% is ~1.31$\times$ wider than 95%.

Gotchas:

• "95% chance the truth is in this interval" = credible-interval language; frequentist coverage is a property of the procedure across repetitions.
• Overlapping CIs on two arms do not imply no significant difference — build the CI on the difference (two 95% CIs can overlap while the difference is significant at 5%).
• Wald proportion intervals fail near $\hat{p} \in \{0, 1\}$ (zero-width at the boundary) — use Wilson or Agresti-Coull for small $n$ or rare events.
• Pooled SE belongs to the null-hypothesis test; unpooled SE belongs to the interval around the observed difference.
• A CI covers the parameter, not future observations — that is a (much wider) prediction interval.