Linear regression

The idea

Linear regression models the target as a weighted sum of features plus noise:

$y = \beta_0 + \beta_1 x_1 + \cdots + \beta_p x_p + \varepsilon, \qquad \hat{\beta} = (X^{\top}X)^{-1}X^{\top}y$

OLS picks the $\hat{\beta}$ minimizing the sum of squared residuals. It is the interview workhorse because every part of it can be probed:

Assumptions — know them in two tiers.

1. For unbiased coefficients: linearity in parameters, and exogeneity ($\mathbb{E}[\varepsilon \mid X] = 0$ — no omitted confounders driving both $x$ and $y$).
2. For valid inference (SEs, t-tests, CIs): independent errors, homoscedasticity (constant error variance), and normal errors for exact small-sample tests. Plus no perfect multicollinearity, or $(X^{\top}X)^{-1}$ does not exist at all.

The distinction is the interview point: heteroscedasticity does not bias $\hat\beta$ — it invalidates the error bars. Prediction survives many violations that kill inference.

Coefficients. $\beta_j$ = expected change in $y$ for a one-unit increase in $x_j$, holding the other features fixed — a partial effect, not a marginal one, which is why coefficients shift when you add or remove correlated features. With $\log(y)$, effects become multiplicative: $e^{\beta_j}$ per unit of $x_j$.

Multicollinearity. Correlated features make $X^{\top}X$ nearly singular: the data cannot tell which of the twins deserves the credit. Detection: $\text{VIF}_j = 1/(1 - R_j^2)$, where $R_j^2$ comes from regressing $x_j$ on the other features; VIF $> 5$-$10$ is the usual alarm, and $\text{SE}(\hat{\beta}_j)$ inflates by $\sqrt{\text{VIF}_j}$. Remedies: drop or combine the redundant features, ridge regression, or PCA. Predictions are mostly fine; coefficient stories are not.

Scoreboards.

$R^2 = 1 - \frac{SS_{res}}{SS_{tot}}, \qquad \bar{R}^2 = 1 - (1 - R^2)\frac{n-1}{n-p-1}$

$R^2$ never drops when you add a feature — even noise — so adjusted $R^2$ (or CV error) is the honest comparator across model sizes. RMSE ($\sqrt{\frac{1}{n}\sum e_i^2}$) amplifies large errors; MAE ($\frac{1}{n}\sum |e_i|$) treats them linearly and resists outliers.

Worked example

Apartment prices (in thousands of dollars) modeled on size and rooms:

$\widehat{\text{price}} = 50 + 0.9 \cdot \text{sqm} + 5 \cdot \text{rooms}$

• Interpretation: among apartments with the same number of rooms, one extra square meter is associated with a $900 higher price. The intercept ($50k at zero size, zero rooms) is an extrapolation artifact, not a real apartment.
• Multicollinearity: sqm and rooms are strongly correlated. Regressing sqm on rooms gives $R_j^2 = 0.90$, so $\text{VIF} = 1/(1 - 0.90) = 10$: the standard error of each twin coefficient is $\sqrt{10} \approx 3.16$ times what it would be with uncorrelated features. Drop rooms, and the sqm coefficient will jump (it absorbs the room effect) — same predictions, different story.
• Adjusted $R^2$: with $n = 50$, $p = 10$ features and $R^2 = 0.60$:

$\bar{R}^2 = 1 - 0.40 \cdot \frac{49}{39} \approx 0.497$

Ten features bought a headline 0.60, but after the parameter tax the explanatory power is closer to half the variance.

• RMSE vs MAE: test residuals $(1, -2, 1, 10)$ in $k give $\text{MAE} = 14/4 = 3.5$ and $\text{RMSE} = \sqrt{(1 + 4 + 1 + 100)/4} = \sqrt{26.5} \approx 5.15$. One bad miss barely moves MAE but pushes RMSE to nearly 1.5x the MAE — exactly the sensitivity you want if large errors are what cost you, and exactly the distortion you do not want if that residual is a data glitch.

Cheat sheet

Estimator: $\hat{\beta} = (X^{\top}X)^{-1}X^{\top}y$ — exists iff no perfect multicollinearity.

Assumptions ladder: linearity + exogeneity → unbiased $\hat\beta$; add independence + homoscedasticity → valid SEs (Gauss-Markov: OLS is BLUE); add normal errors → exact t/F tests.

Metrics: $R^2 = 1 - SS_{res}/SS_{tot}$ (train: never falls with more features; test: can be negative). $\bar{R}^2 = 1-(1-R^2)\frac{n-1}{n-p-1}$. RMSE punishes outliers quadratically; MAE linearly.

• VIF$_j = 1/(1-R_j^2)$; VIF 10 ⇒ SE inflated $\sqrt{10} \approx 3.2$x.
• $\log(y)$ model: coefficient $\beta_j$ ⇒ multiply $y$ by $e^{\beta_j}$ ($\approx 100\beta_j\%$ for small $\beta_j$).
• Multicollinearity hurts inference, not (in-distribution) prediction.
• Gauss-Markov does not require normality — normality is only for exact finite-sample tests.
• Comparing models with different $p$: adjusted $R^2$, AIC/BIC, or CV — never raw train $R^2$.