Feature scaling and preprocessing

The idea

Many models silently assume all features live on a comparable scale. When they do not, the large-scale feature wins by accident of units, not by signal. The mechanism differs by family:

Distance-based models (k-NN, k-means, SVM-RBF): Euclidean distance adds squared differences, so a feature spanning tens of thousands contributes millions to the sum while a feature spanning tens contributes hundreds. The metric becomes a one-feature metric.
Gradient-descent training (linear/logistic via GD, neural nets): unequal scales stretch the loss surface into a ravine with a huge condition number — the learning rate that is stable along the steep axis crawls along the shallow one.
Penalized models (ridge, lasso): the penalty $\lambda\|\beta\|$ charges coefficients uniformly, but the "fair" coefficient size depends on the feature's units. Unscaled, the penalty is a tax assessed in arbitrary currencies.
PCA: components chase variance; the feature with the biggest units owns the first component.

Who is exempt: trees. A split $x_j > t$ only ever compares a feature to its own threshold, so any monotone transform of $x_j$ produces an equivalent tree. This is a real practical advantage of tree ensembles on raw tabular data.

The standard transforms:

\begin{aligned} z &= \frac{x - \mu}{\sigma} && \text{(standardize)}, \\ x' &= \frac{x - x_{\min}}{x_{\max} - x_{\min}} && \text{(min-max)}, \\ x' &= \frac{x - \operatorname{median}(x)}{\operatorname{IQR}(x)} && \text{(robust)}. \end{aligned}

plus $\log(1+x)$ for right-skewed positive features, and one-hot encoding for unordered categories (ordinal codes would fabricate an order that distances and linear models take literally).

The iron rule: fit every transform on the training data only, then apply the frozen parameters everywhere else. The test set must simulate production, and production cannot see the future.

Worked example

k-NN on two features — income (dollars) and age (years). Three people:

Person	Income	Age
A	50,000	25
B	52,000	60
C	60,000	26

Raw Euclidean distances from A:

$d(A, B) = \sqrt{2000^2 + 35^2} \approx 2000.3 \qquad d(A, C) = \sqrt{10000^2 + 1^2} \approx 10000.0$

B is "closest" — a 60-year-old with nearly the same income beats a 26-year-old peer — because a 35-year age gap contributes $35^2 = 1225$ while a modest $10,000 income gap contributes $10^8$ . Age is decorative.

Now standardize with training statistics $\mu_{\text{inc}} = 54{,}000$ , $\sigma_{\text{inc}} = 5{,}000$ , $\mu_{\text{age}} = 37$ , $\sigma_{\text{age}} = 12$ :

Person	$z_{\text{inc}}$	$z_{\text{age}}$
A	$-0.80$	$-1.00$
B	$-0.40$	$1.92$
C	$1.20$	$-0.92$

$d(A, B) = \sqrt{0.40^2 + 2.92^2} \approx 2.95 \qquad d(A, C) = \sqrt{2.00^2 + 0.08^2} \approx 2.00$

The neighborhood flips: C is now A's neighbor. Same data, same algorithm — the only thing that changed is that both features finally speak the same units.

Cheat sheet

Transform	Formula	Use when
Standardize	$(x-\mu)/\sigma$	default for linear/SVM/k-NN/PCA/NN
Min-max	$(x - x_{\min})/(x_{\max}-x_{\min})$	bounded input needed; no big outliers
Robust	$(x-\text{median})/\text{IQR}$	outlier-heavy features
$\log(1+x)$	—	right-skewed positive features (income, counts)
One-hot	indicator per category	unordered categoricals for linear/distance models

Model	Needs scaling?	Mechanism
k-NN, k-means, SVM (RBF)	yes	distances dominated by large-scale features
Linear/logistic via gradient descent, neural nets	yes	ill-conditioned loss surface slows convergence
Ridge / lasso / elastic net	yes	penalty is unit-blind
PCA	yes	variance is unit-dependent
Trees, random forest, gradient boosting	no	splits invariant to monotone transforms
OLS via normal equation	not for predictions	but scaling helps conditioning and coefficient comparability

Fit scaler/imputer/encoder on train only; apply frozen to val/test/prod. Use a pipeline so CV enforces this automatically.
Log first, then standardize — standardizing a skewed feature leaves it skewed.
Min-max is fragile to a single outlier ( $x_{\max}$ moves); robust scaling is not.
One-hot with high-cardinality categories explodes dimensions — consider target encoding (fit within folds! classic leakage source) or grouping rare levels.
Scaling changes coefficients, not predictions, for unpenalized OLS — but changes predictions for anything distance-based or penalized.