The Craft of Training

Where theory meets 3 a.m. debugging

Everything to make a network learn well rather than merely run: this is the practitioner's checklist, and every item exists to solve a specific failure you can now understand.

Initialization: don't start broken

Initialize all weights to zero and every neuron in a layer computes the identical thing — identical gradients, identical updates, forever (the symmetry problem). Initialize too large and activations saturate or explode. The fix: small random values scaled to the layer's size — Xavier/Glorot initialization for sigmoid/tanh, He initialization for ReLU: sample with variance $\frac{2}{n_{\text{in}}}$. Frameworks default to these; know they exist, because a wrong manual init can silently cripple training.

Batch normalization: keep signals in a healthy range

As early layers update, the input distribution to later layers keeps shifting, forcing them to chase a moving target. BatchNorm re-standardizes each layer's pre-activations over the mini-batch (mean 0, variance 1 — statistics path!), then lets the network learn its preferred scale and shift. Effects: allows much higher learning rates, stabilizes deep stacks, mildly regularizes. LayerNorm, its cousin that normalizes across features instead of the batch, is standard in transformers.

Dropout: forced teamwork

During training, randomly zero each hidden unit with probability $p$ (e.g. 0.5); at test time use the full network. No unit can rely on any specific partner, so the network develops redundant, robust features — like training a team where random members skip practice daily, so everyone learns every role. Dropout was key to AlexNet's 2012 breakthrough and remains a cheap, strong regularizer for dense layers.

Learning-rate schedules and optimizers

The learning rate remains the single most important hyperparameter (math path). Modern recipes rarely hold it constant:

• Warmup — start tiny, ramp up over the first steps (protects fragile early training).
• Decay — cosine or step-down schedules: large exploratory steps early, fine footwork near the minimum.

Optimizer default: Adam (or AdamW, which fixes how weight decay interacts with Adam) — per-parameter adaptive steps + momentum. Plain SGD+momentum still sometimes generalizes slightly better in vision; AdamW rules language models.

The debugging liturgy

When (not if) training misbehaves, professionals follow a ritual:

1. Overfit a tiny subset first. A healthy network should reach ~zero loss on 50 examples. If it can't, the bug is in code/data, not capacity.
2. Watch both curves. Training loss flat from step 0 → learning rate/init/data bug. Training falls, validation rises → overfitting: add dropout/augmentation/early stopping.
3. Loss = NaN → learning rate too high or numerical issue (exploding gradients — clip them).
4. Check the data pipeline — mislabeled, unshuffled, or leaking data causes more grief than any architecture choice.

Key takeaways

• He/Xavier init, BatchNorm/LayerNorm, dropout, warmup+decay with AdamW: the standard modern recipe.
• Every trick maps to a failure mode: symmetry, shifting distributions, co-adaptation, unstable steps.
• Debug by overfitting a tiny batch first; read the train/validation curves like vital signs.