Architectures & Transfer Learning

A short history that explains the present

• LeNet-5 (1998) — Yann LeCun's digit reader for bank checks: the conv→pool→dense template, still recognizable in every CNN since.
• AlexNet (2012) — the big bang. 8 layers, trained on two gaming GPUs with ReLU + dropout, it crushed the ImageNet competition (1.2M images, 1,000 classes) by an unheard-of margin. The trio — deep nets, big data, GPUs — has defined AI ever since.
• VGG (2014) — proved that simple uniform 3×3 blocks, stacked deep (16–19 layers), beat clever hand-design. Simplicity + depth became doctrine.
• ResNet (2015) — the enabling trick for real depth. Beyond ~20 layers, plain networks got worse — not from overfitting but because gradients couldn't flow (the vanishing-gradient disease). ResNet's skip connections let each block learn a residual correction to an identity shortcut:

$\mathbf{y} = F(\mathbf{x}) + \mathbf{x}$

Gradients now flow through the "+ x" highway unimpeded; 152-layer networks trained cleanly, surpassing human-level accuracy on ImageNet. Skip connections are so effective they're now everywhere — including inside every transformer (next module).

Since then: EfficientNet (principled scaling of depth/width/resolution) and Vision Transformers (patches as tokens — after the attention lesson you'll understand them for free). Convolution-style models remain excellent, especially at smaller data scales.

Transfer learning: stand on ImageNet's shoulders

The workflow that changed applied vision. A CNN trained on ImageNet has already learned universal visual features — edges, textures, shapes — in its early and middle layers. Those features transfer to your problem, even with only hundreds of images:

1. Download a pretrained network (one line in PyTorch/Keras).
2. Chop off the final classification layer; replace with a fresh head for your classes.
3. Feature extraction: freeze the pretrained body, train only the new head. Fast, works with tiny datasets.
4. Fine-tuning: optionally unfreeze the top few blocks and train them with a small learning rate (don't bulldoze the good features).
5. Always add data augmentation (flips, crops, color jitter) — free regularization.

Rules of thumb: the less data you have and the more similar it is to ImageNet, the more you freeze. A few hundred labeled photos of factory defects, an afternoon, one GPU — 95%+ accuracy is routinely achievable. This pretrain-then-adapt pattern, discovered in vision, is now the organizing principle of all modern AI: foundation models (GPT included) are transfer learning at planetary scale.

Key takeaways

• AlexNet ignited deep learning; VGG standardized depth; ResNet's skip connections made depth trainable.
• $F(\mathbf{x}) + \mathbf{x}$: let layers learn corrections, give gradients a highway.
• Transfer learning (pretrain → replace head → freeze/fine-tune) is the default vision workflow — and the blueprint of the foundation-model era.