Correlation vs causation

The idea

Correlation answers "when I see X, do I tend to see Y?" Causation answers "if I make X happen, does Y change?" Analytics queries answer the first question; product and policy decisions need the second. The gap between them has exactly four inhabitants:

1. Reverse causation — $Y$ drives $X$. (Retained users have more sessions in which to adopt features.)
2. Confounding — some $Z$ drives both. User engagement raises both feature adoption and retention, manufacturing a correlation between them with no causal arrow required.
3. Selection / collider bias — conditioning on a common effect. Analyzing only active users, or only shipped experiments, creates correlations that do not exist in the full population.
4. Chance — especially after scanning many metric pairs; correlation mining at dashboard scale guarantees spurious hits.

The observational data alone cannot distinguish these from real causation — that is not a sample-size problem, it is an identification problem. More rows sharpen the estimate of the wrong quantity.

Randomization is the clean solution: assigning treatment by coin flip severs every arrow from user attributes (measured or not) into treatment. Whatever differs between arms afterwards is treatment effect plus bounded chance. This is precisely why A/B tests are the currency of product decisions, and why "control for everything" is a weaker claim — regression, matching, and propensity scores can only balance confounders you observed, measured well, and thought to include. The unmeasured ones (taste, motivation, engagement) are usually the strongest.

When you cannot randomize, the quasi-experimental ladder, each rung buying credibility with an assumption:

• Difference-in-differences: compare changes in treated vs untreated groups; assumes parallel trends absent treatment.
• Regression discontinuity: compare units just above/below an eligibility cutoff; assumes local comparability.
• Instrumental variables: find something that is relevant (moves treatment), plausibly as-if random, and touches the outcome only through treatment; the usual local-average-effect interpretation also needs monotonicity.
• Matching / propensity scores: compare observably similar units; assumes no unmeasured confounding — the assumption you can never test.

Worked example

Feature adoption vs 90-day retention, 2,000 users. Within each segment, adopters retain 5 points better:

Pooled, adopters retain 37 points worse — Simpson's paradox. The mechanics: adopters are 90% new users (900 of 1,000), while non-adopters are 80% power users (800 of 1,000). The pooled comparison is dominated by who is in each column (user maturity, the confounder), not by what adoption does. Both directions of headline are available from the same table — "adoption cuts retention by 37 points!" and "adoption lifts retention 5 points in every segment!" — which is exactly why an interviewer loves this question.

Which number is right? For "what does adoption do, holding user type fixed?", the stratified +5. But even that is causal only if segment membership is the only confounder — unmeasured engagement within segments can still fake the +5. A randomized encouragement experiment answers the question without the caveat.

The $r = 0$ trap, for completeness: correlation only detects linear association. $X$ symmetric about 0 and $Y = X^2$ gives $\text{cov}(X,Y) = \mathbb{E}[X^3] = 0$ — perfectly dependent, zero correlation.

Cheat sheet

Four rivals to "X causes Y": reverse causation · confounding · selection/collider bias · chance (multiplicity).

Simpson's paradox drill: subgroup direction can reverse in aggregate whenever group composition is unbalanced across a variable linked to the outcome. Decide stratified vs pooled by causal role: confounder → stratify; mediator/consequence of treatment → do not condition on it.

Gotchas:

• $r = 0$ does not imply independence (nonlinear dependence is invisible to Pearson); independence does imply $r = 0$.
• Controlling for a mediator erases the effect you are estimating; controlling for a collider creates bias out of nothing. "Add more controls" is not monotonically safer.
• Self-selected adoption studies are associations, not bounds: confounding can inflate, attenuate, or reverse the causal effect. Treat them as hypotheses and motivation for stronger designs.
• Observational estimate + matching + big n is still observational; credibility comes from design (randomize where possible), triangulation across methods, and sensitivity analysis — not row count.