Dimensionality Reduction & Embeddings

Too many columns

Modern datasets easily carry thousands of features per example. High dimensionality hurts three ways: humans can't visualize it, distance metrics degrade (the curse of dimensionality from the k-NN lesson), and redundant features waste computation while inviting overfitting. Dimensionality reduction compresses features while preserving what matters.

PCA in practice

You met the theory with eigenvalues in the math path; here's the recipe as used daily:

Standardize features (PCA chases variance — unscaled features with big ranges hog it).
Compute principal components — the orthogonal directions of maximal variance (eigenvectors of the covariance matrix).
Look at the explained variance ratio per component. Keep enough components for, say, 95% of total variance.
Project the data onto those components: 1,000 features → maybe 50, most information intact.

Each component is a weighted mix of original features — sometimes interpretable ("component 1 ≈ overall size", "component 2 ≈ luxury vs. budget"), often not. PCA is linear and unbeatable as a fast preprocessing and compression step, but it can't unroll curved structure.

Seeing your data: t-SNE and UMAP

For visualization, two nonlinear methods dominate. t-SNE and UMAP squash high-dimensional data to 2-D while trying to keep neighbors together. Feed them images of handwritten digits and, with no labels at all, ten distinct islands appear — one per digit. It feels like magic and is the standard first plot for embeddings.

Read such plots with discipline:

Neighborhoods are meaningful; distances between far-apart clusters are not.
Cluster sizes and shapes are artifacts of the algorithm.
Different random seeds and hyperparameters (perplexity for t-SNE, n_neighbors for UMAP) rearrange the map.

Use them to explore and hypothesize, never as quantitative evidence. UMAP is faster and better preserves global structure — the modern default.

The embedding worldview

Dimensionality reduction points at a grander idea that dominates modern AI: represent anything — words, images, songs, users, proteins — as a modest-length vector where geometry encodes meaning: similar things sit close, and directions can capture concepts. These learned representations are called embeddings. Recommendation ("users near you liked…"), semantic search, RAG systems for LLMs, face recognition — all are geometry in embedding space. The deep learning path shows how such embeddings are learned rather than hand-constructed.

Key takeaways

PCA: standardize → keep components covering ~95% variance → project. Linear, fast, the workhorse.
t-SNE/UMAP make stunning 2-D maps — trust neighborhoods, distrust global distances and sizes.
Embeddings — meaning as geometry — are the conceptual bridge from classic ML to modern AI.