Layers, Depth & Representation

Assembling neurons into networks

A feed-forward network (multilayer perceptron, MLP) arranges neurons in layers:

• Input layer — the raw feature vector (e.g. 784 pixel values for a 28×28 digit image).
• Hidden layers — each neuron in a layer connects to every output of the previous layer ("fully connected" / "dense").
• Output layer — e.g. 10 neurons + softmax for digit classification.

One layer's computation is compact in matrix form — this is why linear algebra is the language of DL:

$\mathbf{a}^{(l)} = g\!\left(W^{(l)} \mathbf{a}^{(l-1)} + \mathbf{b}^{(l)}\right)$

The whole network is these lines composed — a function machine feeding a function machine (math path, lesson 2). Count the parameters: a 784→128→64→10 network has $784{\cdot}128 + 128{\cdot}64 + 64{\cdot}10 + \text{biases} \approx 109{,}000$ learnable numbers. GPT-class models: hundreds of billions. Same equation.

What hidden layers actually learn

Here is the profound part. Train a digit classifier and inspect the layers:

• Layer 1 neurons respond to tiny primitives: edges at particular angles, small strokes.
• Layer 2 combines edges into motifs: curves, corners, loops.
• Layer 3 combines motifs into parts: "top circle", "diagonal tail".
• Output combines parts into concepts: "that's an 8".

Nobody programmed edges or loops. The network invented a hierarchy of features because the hierarchy helps minimize loss. This is representation learning, and it's the answer to the feature-engineering toil of the ML path: for images, audio and text, the network engineers its own features — better than humans ever managed. In 2012, a deep network (AlexNet) halved the error rate of hand-engineered vision pipelines overnight; that event ignited the modern AI era.

Width vs. depth

The universal approximation theorem says one hidden layer, wide enough, can approximate any continuous function. So why go deep? Efficiency: depth builds features compositionally — reusing edges in all corners, corners in all shapes — while a shallow-wide network must relearn every combination from scratch. Functions with hierarchical structure (i.e. the real world: pixels→objects, characters→meaning) need exponentially fewer neurons when expressed deep. Depth mirrors the structure of reality; that's why it wins.

Sizing in practice

More depth/width = more capacity = more overfitting risk — the bias-variance story again, now with millions of parameters. The modern workflow: pick a proven architecture size for your data scale, then control overfitting with regularization (dropout, early stopping, augmentation) rather than timidly shrinking the network.

Key takeaways

• A network is stacked matrix-multiplications with nonlinearities: $\mathbf{a}^{(l)} = g(W\mathbf{a}^{(l-1)} + \mathbf{b})$.
• Hidden layers learn a feature hierarchy automatically — representation learning replaced hand-crafted features for perception.
• Depth beats width because the world is compositional.