Eigenvalues & Why Dimensions Get Reduced

The special directions of a matrix

Apply a transformation matrix to lots of vectors and most of them get knocked off their original direction. But a few special vectors merely get stretched or shrunk, staying on the line they started on. These are the eigenvectors of the matrix, and their stretch factors are the eigenvalues:

$A\mathbf{v} = \lambda \mathbf{v}$

Read: "applying $A$ to $\mathbf{v}$ is the same as just scaling $\mathbf{v}$ by the number $\lambda$."

Think of a spinning globe: every point moves, except the axis of rotation. That axis is an eigenvector. Eigenvectors reveal the "natural axes" of a transformation — the directions along which its behavior is simplest.

Why should an AI learner care?

1. Principal Component Analysis (PCA). Real datasets have redundant features (a house's size in m² and in ft² carry the same information). PCA finds the directions along which the data actually varies — which turn out to be eigenvectors of the data's covariance matrix. Keep the few directions with large eigenvalues, discard the rest, and you've compressed 1,000 features down to 50 while losing little information. This is the classic tool for dimensionality reduction, visualization, and noise removal.

2. Understanding training dynamics. Whether gradient descent converges smoothly or oscillates depends on the eigenvalues of certain matrices built from the model. When you hear "ill-conditioned problem", that's eigenvalue-speak for "some directions are stretched vastly more than others".

3. Google's original PageRank was an eigenvector computation: the importance of every page on the web is the leading eigenvector of the link matrix.

Decompositions: taking matrices apart

Just as 12 factors into 2·2·3, matrices factor into simpler pieces:

• Eigendecomposition: $A = Q\Lambda Q^{-1}$ — a rotation, a pure stretch along the eigen-directions ($\Lambda$ is diagonal, holding eigenvalues), and a rotation back.
• Singular Value Decomposition (SVD): works for any rectangular matrix — the swiss-army knife behind recommender systems and data compression. Netflix-style recommendations famously build on this idea: factor the (users × movies) rating matrix into compact "taste" factors.

You don't need to compute these by hand — libraries do it in one call. What matters is the intuition: decompositions reveal the simple structure hiding inside a complicated matrix.

Key takeaways

• Eigenvectors are directions a matrix only stretches; eigenvalues are the stretch factors.
• PCA = keep the high-variance eigen-directions, drop the rest → fewer dimensions, most information kept.
• Decompositions (eigen, SVD) factor matrices into interpretable parts and power compression and recommenders.