Embeddings

The idea

Neural networks compute with continuous vectors, but much of the world is discrete: words, users, products, categories. The naive encoding — one-hot vectors — is dimension-bloated (one axis per item) and, worse, geometry-free: every pair of one-hots is orthogonal, so cat is exactly as far from dog as from carburetor, and nothing learned about one item can transfer to a similar one.

An embedding replaces each item with a learned dense vector of modest dimension ($d$ from tens to a few thousand), stored as rows of a matrix $E \in \mathbb{R}^{V \times d}$. "Looking up an embedding" is just indexing row $i$ (equivalently, multiplying a one-hot by $E$). The rows are ordinary parameters, trained by backprop against whatever loss the model has.

The magic is why geometry emerges: the loss can only fall if items that behave similarly under the training objective — words in similar contexts, products bought by similar users — get vectors the rest of the network can treat similarly, i.e. nearby vectors. Nobody assigns meaning to axes; similar usage → similar vectors is an optimization consequence. word2vec made this famous with pure self-supervision (skip-gram: predict context words from the center word), yielding geometry clean enough that $king - man + woman$ lands nearest queen.

Similarity is measured with cosine,

$\cos(u, v) = \frac{u \cdot v}{\lVert u \rVert\, \lVert v \rVert} \in [-1, 1]$

when only direction should matter (norms often encode frequency), or the raw dot product when magnitude is genuine signal. Two generational upgrades to know: static → contextual (a transformer encoder gives each token a vector that depends on its sentence, splitting the senses of "bank"), and word → sentence/document embeddings (pooled contextual vectors trained contrastively), which power semantic search and retrieval-augmented generation.

Worked example

Toy 2-d embeddings: $\mathit{cat} = (0.9, 0.2)$, $\mathit{dog} = (0.85, 0.3)$, $\mathit{car} = (0.1, 0.95)$.

For cat versus dog:

Cosine of cat vs car: dot $= 0.09 + 0.19 = 0.28$, $\lVert \mathit{car} \rVert \approx 0.955$, giving $0.28 / (0.922 \times 0.955) \approx 0.32$. The animals are nearly parallel (0.99); animal-vs-vehicle is weakly related (0.32). Distances now mean something — the entire point, in two dimensions.

And that is already a retrieval system: given the query cat, ranking all other vectors by cosine returns dog first and car last. Semantic search over a million documents is this exact computation with $d$ in the hundreds and an approximate nearest-neighbor index (HNSW, IVF) standing in for the brute-force scan once exhaustive comparison gets too slow.

Scale check for real systems: a 50k-word vocabulary at $d = 768$ is a $50{,}000 \times 768$ table ≈ 38.4M parameters — often the largest single layer in a small model, and the reason embedding tables get tied between input and output, quantized, or hashed in production recsys.

Cheat sheet

• Embedding layer: matrix $E$, shape $V \times d$; lookup = row index. Plain untied lookups have sparse row updates, but weight tying and other objectives can make gradients denser.
• Cosine: dot product over the product of norms — direction only. Dot product: direction × magnitude. Normalize vectors once at index time → cosine becomes plain dot/inner-product search.
• Typical dimensions: word2vec/GloVe 100-300; BERT tokens 768; sentence embedding APIs 384-3072; recsys 32-256.
• Getting embeddings: (1) train an embedding layer end-to-end with your task; (2) self-supervised pretraining (skip-gram, masked LM); (3) contrastive learning on pairs (sentence encoders, CLIP); (4) hidden states of a pretrained model.
• Uses: k-NN semantic search / retrieval (ANN index: HNSW, IVF), clustering, dedup, features for downstream models, cold-start via content encoders.
• Evaluation: nearest-neighbor spot checks, clustering coherence, retrieval recall@k / MRR on labeled pairs, downstream probe accuracy.

Gotchas

• Vectors from different models (or different training runs) live in incompatible spaces — never mix; re-embed everything on upgrade.
• Large norms can dominate dot-product retrieval; hubness can also arise from high-dimensional geometry and may persist under cosine. Diagnose before choosing normalization or debiasing.
• Anisotropy: raw transformer token vectors bunch into a narrow cone; pooled-without-training sentence vectors are mediocre — use encoders trained for similarity.
• Averaging word vectors loses order and negation ("not good" ≈ "good").
• Embedding tables dominate params at small $d_{\text{model}}$ — tie input/output embeddings, or hash/quantize in recsys.