Maximum likelihood estimation

The idea

You have data and a family of models indexed by a parameter $\theta$. Every candidate $\theta$ assigns the observed data some probability. Maximum likelihood says: choose the $\theta$ that made what you saw least surprising.

$\hat{\theta}_{\text{MLE}} = \arg\max_\theta \; L(\theta) = \arg\max_\theta \prod_{i=1}^n p(x_i \mid \theta)$

In practice you always work with the log-likelihood $\ell(\theta) = \sum_i \log p(x_i \mid \theta)$: the product becomes a sum, underflow disappears, derivatives are clean, and the argmax is unchanged. The recipe is mechanical: write $\ell$, differentiate, set to zero, check the second derivative is negative.

Three reasons this one idea keeps reappearing:

1. It recovers the obvious estimators. The MLE of a Bernoulli $p$ is the sample proportion; of a Poisson $\lambda$ or Gaussian $\mu$, the sample mean. The principled machinery agrees with intuition on easy cases, which is what licenses trusting it on hard ones.
2. It is what your loss functions secretly are. Least squares is MLE under Gaussian noise; cross-entropy is the negative Bernoulli/categorical log-likelihood; Poisson regression's loss is the negative Poisson log-likelihood. "Pick a noise model, take negative log" generates most of the standard losses.
3. It has strong guarantees — asymptotically. MLEs are consistent (converge to the truth), asymptotically normal (enabling standard errors and Wald tests), asymptotically efficient (no unbiased estimator does better in the limit), and invariant (MLE of $g(\theta)$ is $g(\hat{\theta})$).

The fine print: those guarantees are large-sample. In small samples MLEs can be biased — the Gaussian variance MLE divides by $n$ instead of $n-1$ — and with sparse data they hit degenerate boundary answers, like estimating a probability of exactly zero because an event never occurred in 20 trials. The standard fix is a prior: MAP estimation maximizes likelihood $\times$ prior, and its negative log is exactly "loss + regularizer".

Worked example

Ten independent trials, seven successes. Model: Bernoulli$(p)$.

$\ell(p) = 7 \log p + 3 \log(1 - p)$

Differentiate and solve:

$\ell'(p) = \frac{7}{p} - \frac{3}{1-p} = 0 \;\Rightarrow\; 7(1-p) = 3p \;\Rightarrow\; \hat{p} = 0.7$

Numeric sanity check (natural logs):

$p = 0.7$ wins, and note the asymmetry: 0.9 is worse than 0.5 despite being the same distance from 0.7, because those three failures are very improbable under $p = 0.9$ — the $\log$ of a small probability is savagely negative. That asymmetry is the same mechanism that makes cross-entropy punish a confident wrong prediction so much harder than a hedged one.

Cheat sheet

Recipe: write $\ell(\theta) = \sum_i \log p(x_i \mid \theta)$ → solve $\ell'(\theta) = 0$ → verify $\ell''(\hat{\theta}) < 0$.

Loss ↔ likelihood dictionary:

• Squared error ↔ Gaussian noise
• Cross-entropy / log loss ↔ Bernoulli or categorical likelihood
• Absolute error ↔ Laplace noise
• L2 penalty ↔ Gaussian prior (MAP); L1 penalty ↔ Laplace prior (MAP)

Properties (large $n$): consistent · asymptotically normal · asymptotically efficient (Cramer-Rao) · invariance: $\widehat{g(\theta)} = g(\hat{\theta})$.

Gotchas:

• Small-sample bias: variance MLE divides by $n$; use $n-1$ for unbiased.
• Zero-count boundary estimates ($\hat{p} = 0$) break downstream multiplication — smooth with a prior (Laplace/add-one, or Beta prior).
• Likelihood is a function of $\theta$ with data fixed; it is not a probability distribution over $\theta$ — it need not integrate to 1, and turning it into $P(\theta \mid \text{data})$ requires Bayes and a prior.
• MLE properties assume the model family contains (or nearly contains) the truth; under gross misspecification the MLE converges to the best in-family approximation, not the truth.