Matrices & Matrix Multiplication

Two ways to see a matrix

A matrix is a rectangular grid of numbers, like a spreadsheet:

$A = \begin{bmatrix} 2 & 0 \\ 1 & 3 \end{bmatrix}$

This one has 2 rows and 2 columns (a "2×2 matrix"). There are two equally important ways to think about matrices:

1. A matrix is a table of data. Each row is one example (a house), each column one feature (size, age). A dataset of 10,000 houses with 20 features is a 10,000 × 20 matrix. When ML engineers say "design matrix", they mean this.

2. A matrix is a function that transforms vectors. Multiplying a vector by a matrix rotates, stretches, flips or squashes it. Every layer of a neural network is a matrix waiting to transform whatever vector arrives.

Matrix–vector multiplication

To compute $A\mathbf{x}$, take the dot product of each row of $A$ with $\mathbf{x}$:

$\begin{bmatrix} 2 & 0 \\ 1 & 3 \end{bmatrix} \begin{bmatrix} 1 \\ 2 \end{bmatrix} = \begin{bmatrix} 2\cdot1 + 0\cdot2 \\ 1\cdot1 + 3\cdot2 \end{bmatrix} = \begin{bmatrix} 2 \\ 7 \end{bmatrix}$

A vector went in, a (transformed) vector came out — a matrix is a function. A neural network layer computes exactly

$\mathbf{y} = W\mathbf{x} + \mathbf{b}$

where $W$ holds the layer's weights and $\mathbf{b}$ its biases. Notice this is the multi-dimensional version of $y = wx + b$ from lesson one. The pattern never changes; only the dimensions grow.

Matrix–matrix multiplication

Multiplying matrix $A$ (m×n) by matrix $B$ (n×p) gives an m×p matrix: entry $(i,j)$ is the dot product of row $i$ of $A$ with column $j$ of $B$. Two crucial properties:

• Inner dimensions must match — (m×n)(n×p) works; the n's must agree.
• Order matters — $AB \neq BA$ in general. Transformations applied in a different order give different results (rotate-then-stretch ≠ stretch-then-rotate).

Composing two transformations = multiplying their matrices. A 50-layer network is (roughly) 50 matrix multiplications chained together. Training GPT-class models is, computationally, trillions of these operations — which is why the entire AI industry runs on GPUs, chips originally built to multiply matrices for 3-D graphics.

Special matrices worth knowing

• Identity $I$: ones on the diagonal, zeros elsewhere; $I\mathbf{x} = \mathbf{x}$ (does nothing).
• Transpose $A^T$: flip rows and columns.
• Inverse $A^{-1}$: the "undo" matrix, $A^{-1}A = I$ (when it exists).

Key takeaways

• Matrix = data table and vector transformer; both views matter.
• A neural layer is $W\mathbf{x} + \mathbf{b}$ — matrices are the weights of AI.
• Matrix multiplication composes transformations; order matters.