Logistic regression

The idea

Logistic regression is a linear model whose output is squashed into a valid probability by the sigmoid:

$p = \sigma(z) = \frac{1}{1 + e^{-z}}, \qquad z = \beta_0 + \beta^{\top}x$

The squashing hides the linearity, but it is still there — solve for $z$ and the model is linear in the log-odds:

$\log\frac{p}{1-p} = \beta_0 + \beta^{\top}x$

That single line answers three classic probes:

• Coefficient meaning: $\beta_j$ is the change in log-odds per unit of $x_j$; equivalently, the odds multiply by $e^{\beta_j}$ (the odds ratio). The probability change is not constant — it is largest near $p = 0.5$ where the sigmoid is steep (slope $p(1-p)$, max $0.25$) and tiny near 0 or 1.
• Decision boundary: thresholding at $p \ge 0.5$ is the same as $z \ge 0$, a hyperplane. Logistic regression is a linear classifier; the sigmoid changes the scores, not the geometry.
• Loss: with Bernoulli labels, the likelihood of the data is $\prod_i p_i^{y_i}(1-p_i)^{1-y_i}$. Taking negative logs gives binary cross-entropy:

$L(\beta) = -\sum_i \Big[ y_i \log p_i + (1 - y_i)\log(1 - p_i) \Big]$

So the loss is not a design choice — it is maximum likelihood.

Why not MSE? Two independent reasons interviewers want to hear. Optimization: MSE through the sigmoid is non-convex in $\beta$, and its per-example gradient carries a factor $\sigma'(z) = p(1-p)$ that vanishes when the model is saturated — a prediction of 0.999 against a true label 0 is confidently wrong yet learns almost nothing. Statistics: cross-entropy is the Bernoulli negative log-likelihood and a proper scoring rule, so minimizing it pushes toward calibrated probabilities. Its gradient is beautifully clean — the sigmoid derivative cancels:

$\nabla_\beta L = \sum_i (p_i - y_i)\, x_i$

Error times input; no vanishing factor.

Failure mode to know: with perfectly separable classes, the unregularized log-likelihood approaches its finite supremum as $\lVert\beta\rVert \to \infty$ but has no finite maximizer, so fitted weights diverge. Regularization (L2 by default in most libraries) restores a finite, stable solution.

Worked example

Model for promotion within two years from years of experience:

$\log\frac{p}{1-p} = -3 + 0.8 \cdot \text{years}$

For a person with $\text{years} = 5$: $z = -3 + 4 = 1$, so

$p = \sigma(1) = \frac{1}{1 + e^{-1}} \approx 0.731$

and the odds are $p/(1-p) = 0.731/0.269 \approx 2.72 = e^{1}$ — the odds are literally $e^{z}$, which is the whole trick of the log-odds scale.

One more year ($\text{years} = 6$): odds multiply by $e^{0.8} \approx 2.23$, giving odds $2.72 \times 2.23 \approx 6.05$ and $p = 6.05/7.05 \approx 0.858$. So the same +1 year moved probability by about $+0.13$ here — but a person at $\text{years} = 1$ ($z = -2.2$, $p \approx 0.10$) moving to $z = -1.4$ reaches only $p \approx 0.20$: identical odds ratio, different probability jump. Never report an odds ratio as "the probability increases by."

Cheat sheet

Model: $p = \sigma(\beta_0 + \beta^{\top}x)$; $\log\frac{p}{1-p}$ linear in $x$. Boundary: $z = 0$ hyperplane.

Loss (= Bernoulli NLL): $-\sum \left[ y\log p + (1-y)\log(1-p) \right]$; gradient $\sum (p_i - y_i)x_i$; convex — GD/Newton converge globally.

• Why not MSE: non-convex through sigmoid + vanishing gradient when confidently wrong; cross-entropy is MLE, convex, gradient $\propto$ error.
• Coefficients are log-odds effects; $e^{\beta}$ is an odds ratio, not a probability change or a risk ratio (they coincide only when $p$ is small).
• Perfect separation ⇒ divergent weights ⇒ regularize (sklearn penalizes with L2 by default — remember that when comparing to statsmodels output).
• Needs scaled features for gradient-based fitting and for penalized variants; the unpenalized optimum itself is scale-equivariant.
• Multiclass: softmax / multinomial logistic; one-vs-rest is the simpler fallback.
• Outputs are probabilities worth calibration-checking — log loss and reliability curves, not just accuracy.