Classification metrics

The idea

Every hard classification decision lands in one cell of the confusion matrix:

All the named metrics are just ratios over this table, each answering a different question:

• Precision — alert quality: when the model shouts, is it right?
• Recall — coverage: of the real positives, how many did we catch?
• Accuracy — overall correctness, which silently assumes classes are balanced and errors cost the same. With a 97% negative base rate, the all-negative model scores 97% — check the majority baseline before quoting accuracy, ever.

Precision and recall fight through the threshold: flag more (lower threshold) and recall rises while precision drifts down toward the base rate. F1 compresses the pair into one number using the harmonic mean,

$F_1 = 2\,\frac{P \cdot R}{P + R},$

which is dominated by the weaker of the two — you cannot F1-launder a degenerate classifier ($P = 1.0, R = 0.02$ gives $F_1 \approx 0.04$, not $0.51$). $F_\beta$ generalizes: $\beta > 1$ weights recall more ($F_2$ for screening), $\beta < 1$ weights precision.

ROC-AUC evaluates the scores rather than one thresholded decision. The ROC curve plots TPR against FPR as the threshold sweeps; the area under it equals the probability that a random positive outranks a random negative. AUC 0.5 = coin-flip ranking, 1.0 = perfect separation. It is threshold-free and invariant to monotone score transforms — which is both its power (fair model comparison) and its blind spot (says nothing about calibration or about performance at your operating point, and can look rosy under heavy imbalance because FPR divides by a huge negative count).

Worked example

A disease classifier on $n = 1000$ patients, prevalence 6% (60 positives):

$\text{Accuracy} = \frac{40 + 930}{1000} = 0.97 \qquad \text{FPR} = \frac{10}{940} \approx 0.011$

The readings that matter:

• Accuracy 0.97 sounds superb — but all-negative scores 0.94. The model's entire contribution lives in that 3-point gap; precision/recall expose it directly.
• Precision 0.80: 1 in 5 alarms is false. Recall 0.667: a third of true cases walk out undetected — likely the number a medical stakeholder cares about most.
• Raising the threshold would trade recall down for precision up; whether that is good depends entirely on the cost of an FN (missed disease) vs an FP (unnecessary follow-up test) — the matrix cannot decide that for you.

Cheat sheet

From the matrix: $P = \frac{TP}{TP+FP}$, $R = \text{TPR} = \frac{TP}{TP+FN}$, $\text{FPR} = \frac{FP}{FP+TN} = 1 - \text{specificity}$, $F_1 = \frac{2PR}{P+R}$, $F_\beta = (1+\beta^2)\frac{PR}{\beta^2 P + R}$.

• ROC-AUC $= P(\text{score}_+ > \text{score}_-)$ for random +/− pair; threshold-free; monotone-invariant; 0.5 = random.
• Threshold up ⇒ recall down, precision (typically) up; threshold down ⇒ recall → 1, precision → prevalence.
• Accuracy floor to beat: $\max(\pi, 1-\pi)$ for prevalence $\pi$.
• F1 ignores TN entirely — sensible for needle-in-haystack tasks, misleading when true negatives are what you value.
• Harmonic-mean trap: $P = 1.0$, $R = 0.02$ ⇒ $F_1 \approx 0.04$.
• Report the operating point WITH the metric: "P/R at threshold t," never bare F1 with an unstated default threshold.