CLT and sampling distributions

The idea

Compute a statistic from a random sample and you get a number. Draw a different sample and you get a different number. The sampling distribution is the distribution of that statistic across all the samples you might have drawn — it is the honest answer to "how much should I trust this estimate?" Inference is essentially the art of reasoning about a distribution you only ever see one draw from.

For the sample mean, the central limit theorem gives the sampling distribution almost for free. If $X_1, \dots, X_n$ are i.i.d. with mean $\mu$ and finite variance $\sigma^2$ :

$\bar{X}_n \;\approx\; \mathcal{N}\!\left(\mu, \; \frac{\sigma^2}{n}\right) \quad \text{for large } n$

Two separate miracles are packed in here:

Normality regardless of input shape. The raw data can be binary, skewed, multimodal — averages of enough of it are bell-shaped anyway. This is why the same $z$ / $t$ machinery serves conversion rates and revenue alike.
The $\sqrt{n}$ law. The spread of the sampling distribution — the standard error $\text{SE} = \sigma/\sqrt{n}$ — shrinks with the square root of sample size. Precision is bought at quadratic cost: half the error needs $4\times$ the data, a tenth needs $100\times$ .

Keep the two spreads straight: the SD ( $\sigma$ ) is about individuals and never shrinks with more data; the SE is about the estimate and does. For a proportion, $\text{SE}(\hat{p}) = \sqrt{p(1-p)/n}$ — a special case since a proportion is just a mean of 0/1 values.

"Large $n$ " depends on skew: symmetric data is effectively normal by $n \approx 30$ ; heavily skewed metrics like revenue may need thousands. And the two load-bearing assumptions — independence and finite variance — are exactly where real data attacks: repeated events per user are correlated (cluster by user, or your SE is fake), and whale-dominated metrics converge painfully slowly.

Worked example

Revenue per user: true mean $\mu = 50$ dollars, SD $\sigma = 100$ dollars (right-skewed; most users spend nothing). Sample $n = 10{,}000$ users:

$\text{SE} = \frac{100}{\sqrt{10000}} = 1$

So $\bar{X} \approx \mathcal{N}(50, 1^2)$ : about 95% of such samples yield a mean in $[48, 52]$ . Now shrink the sample:

$n$	SE $= 100/\sqrt{n}$	95% range for $\bar{X}$
100	$10	$30 – $70
1,000	$3.16	$43.8 – $56.2
10,000	$1	$48 – $52
40,000	$0.50	$49 – $51

Note the diminishing returns: going from 10,000 to 40,000 users (4x the data, 4x the cost) only halves the interval. And an A/B test comparing two arms needs the SE of a difference, $\sqrt{\text{SE}_A^2 + \text{SE}_B^2}$ , which is $\sqrt{2}$ larger than either arm alone — one reason experiments always need more data than intuition suggests.

Cheat sheet

CLT: $\bar{X}_n \approx \mathcal{N}(\mu, \sigma^2/n)$ for i.i.d. data with finite variance; equivalently $\frac{\bar{X}_n - \mu}{\sigma/\sqrt{n}} \approx \mathcal{N}(0,1)$ .
SE of mean: $s/\sqrt{n}$ . SE of proportion: $\sqrt{\hat{p}(1-\hat{p})/n}$ (max $0.5/\sqrt{n}$ at $\hat{p}=0.5$ ).
SE of difference of independent estimates: $\sqrt{\text{SE}_1^2 + \text{SE}_2^2}$ .
Precision law: SE $\propto 1/\sqrt{n}$ → halve the error = $4\times$ the data.
SD describes individuals (never shrinks); SE describes the estimator (shrinks with $n$ ).

Symptom	Problem	Fix
Metric dominated by whales	slow CLT convergence, unstable means	winsorize/cap, log-transform, compare quantiles, or bootstrap
Multiple rows per user	dependence → SE too small, false positives	aggregate to one row per randomization unit, or cluster-robust SEs
Small $n$ + skew	normal approx poor	$t$ -distribution helps only with symmetry — prefer bootstrap
Need tail quantities (p99, max)	CLT says nothing about tails	extreme value methods or direct quantile estimation

The CLT applies to sums and means. Sample medians and quantiles have their own (different) asymptotics; the sample maximum is not normal at all.
Rule of thumb: $n \geq 30$ for mild skew; heavily skewed metrics can need $n$ in the thousands before the normal approximation is trustworthy.