Class imbalance · Math Foundry

The idea

With a 1% positive rate, the model can be 99% accurate by doing nothing. Imbalance breaks the silent assumptions of default classification — that 0.5 is a sensible threshold, that accuracy measures usefulness, and that both error types cost the same. The fixes live at three layers:

1. The metric layer. Accuracy → precision/recall and PR curves. The mechanical reason ROC flatters: FPR $= FP/(FP+TN)$ has the giant negative class in its denominator, so an avalanche of false alarms barely dents it, while precision $= TP/(TP+FP)$ takes the hit in full. Judge PR-AUC against its no-skill baseline — the prevalence, not 0.5. Use stratified CV so folds contain positives at all.

2. The training layer. Make rare examples matter:

Class weights — count each minority example $w$ times in the loss (typically inverse frequency). Same effect as oversampling, no duplication, one line in most libraries.
Random over/undersampling — duplicate minority rows or discard majority rows. Undersampling can waste data; oversampling can overfit duplicated points.
SMOTE — synthesize minority points by interpolating between minority neighbors. Helps sometimes on tabular data; hurts when classes overlap (it fabricates positives inside majority territory). And it must run inside the CV fold — SMOTE before the split leaks and inflates scores, the single most common imbalance bug.

3. The decision layer — usually the right one. Imbalance is really a cost asymmetry problem. With calibrated probability $p$ , a missed-positive cost $C_{FN}$ , and a false-alarm cost $C_{FP}$ that applies only when the case is actually negative, flag when expected cost says so:

$p \cdot C_{FN} > (1-p) \cdot C_{FP} \quad\Longleftrightarrow\quad p > \frac{C_{FP}}{C_{FP} + C_{FN}}$

If taking the action has a fixed cost $C_{action}$ for every flagged case (such as a manual review), compare $C_{action}$ directly with $pC_{FN}$ : flag when $p > C_{action}/C_{FN}$ .

The 0.5 default corresponds to equal costs — almost never true when one class is rare. A well-ranked, well-calibrated model plus the right threshold often beats any resampling scheme, and it keeps probabilities honest for downstream use. Caveat for rebalancers: training on altered prevalence distorts the posterior — recalibrate on true-prevalence data or apply the known prior correction.

Threshold lab

Move one decision threshold. The model scores stay fixed; every metric and cost changes around that choice.

Score 0Score 1

0.50

Actual positiveActual negative

Threshold: 0.50Positive prevalence: 10%

270

630

Precision

20.6%

Recall

70%

31.8%

Accuracy

70%

Cost / 1k

$17,700

Cost assumes $10 per false alarm and $500 per miss. Lower thresholds catch more positives; prevalence determines how many false alarms surround them.

Worked example

Fraud detection: $10{,}000$ transactions, prevalence 1% → 100 fraud, 9,900 legitimate. Model operates at TPR $= 0.80$ , FPR $= 0.05$ :

$TP = 80, \quad FN = 20, \quad FP = 0.05 \times 9900 = 495, \quad TN = 9405$

$\text{Precision} = \frac{80}{80 + 495} = \frac{80}{575} \approx 0.139$

On the ROC plot, (FPR 0.05, TPR 0.80) looks like a strong classifier. In the queue a human reviews, 6 of every 7 alerts are false — the ROC point and the analyst experience are describing the same model.

Now the cost view. A missed fraud costs $500; every manual review costs $10:

$\text{threshold} = \frac{C_{review}}{C_{FN}} = \frac{10}{500} = 0.02$

Flag anything with fraud probability above 2% — twenty-five times more aggressive than the 0.5 default. Sanity-check with expected cost at this operating point: the current confusion matrix costs $20 \times 500 + 575 \times 10 = 15{,}750$ dollars per 10k transactions. A blanket-review policy would cost $10{,}000 \times 10 = 100{,}000$ dollars; doing nothing costs $100 \times 500 = 50{,}000$ dollars. The model earns its keep only through where you put the threshold — and 0.5 was never a candidate.

Cheat sheet

Cost-optimal threshold (calibrated $p$ ): with a false-alarm-only cost, flag when $p > \frac{C_{FP}}{C_{FP}+C_{FN}}$ ; with a fixed action/review cost on every alert, flag when $p > C_{action}/C_{FN}$ . Equal false-positive/false-negative costs → 0.5; rare-and-costly misses → tiny threshold.

PR vs ROC: rare positives + flagging workload ⇒ PR (baseline = prevalence). Balanced classes / prevalence-free comparison ⇒ ROC (baseline 0.5).

Tool	What it does	Watch out
Threshold tuning	moves operating point, model untouched	needs calibrated or at least monotone scores
Class weights	reweights loss toward minority	distorts calibration; recalibrate after
Random oversampling	duplicates minority rows	overfits duplicates
Random undersampling	drops majority rows	discards signal; try ensembles of undersamples
SMOTE	interpolates synthetic minority points	leaks if before split; fabricates points in overlap regions
Cost-sensitive learning	bakes $C_{FP}, C_{FN}$ into training	requires actually knowing the costs

Fit resampling INSIDE the CV fold; evaluate ALWAYS on untouched, true-prevalence data.
No-skill precision = prevalence; quote PR-AUC relative to it.
Rebalanced training ⇒ probabilities biased toward minority; fix via calibration on true-prevalence data or logit shift by $\log(\text{odds}_{\text{deploy}} / \text{odds}_{\text{train}})$ .
Stratify CV folds; never let a fold go positive-free.
Alert-budget framing: capacity of $k$ reviews/day ⇒ optimize precision@k / recall@k, not a global scalar.
First question to ask: are the negatives clean, or just undetected positives? Label noise masquerades as imbalance.