Batch normalization
Batch normalization is a technique that normalises the activations of each layer across the current mini-batch during training. For each feature dimension,...
Stabilising Training with Batch Normalization
Batch normalization is a technique that normalises the activations of each layer across the current mini-batch during training. For each feature dimension, it subtracts the batch mean and divides by the batch standard deviation, then applies learned scale and shift parameters called gamma and beta. This keeps activations in a stable numerical range throughout training.
The primary benefit is faster and more stable training. Without normalisation, activations can grow or shrink uncontrollably as they pass through many layers, a problem called internal covariate shift. Batch normalisation keeps each layer's input distribution consistent across training steps, allowing higher learning rates and reducing sensitivity to weight initialisation.
Batch normalisation also has a mild regularising effect. Because the normalisation statistics are computed per mini-batch rather than across the full dataset, each example is normalised using slightly different statistics depending on which other examples are in the batch. This stochasticity acts similarly to dropout, adding a small amount of noise that discourages overfitting.
At inference time, the per-batch mean and variance are replaced by running averages accumulated during training, so the transformation is deterministic and independent of batch composition. This is a critical implementation detail: forgetting to switch batch norm to evaluation mode at inference leads to inconsistent predictions.
A concrete example: training a very deep network like a 50-layer ResNet without batch normalisation is nearly impossible with gradient-based methods because activations collapse or explode before the signal reaches early layers. Adding batch normalisation after every convolutional layer makes training stable and fast, enabling accuracies that were otherwise unachievable.
Exam tip: distinguish batch normalisation from layer normalisation. Batch normalisation normalises across the batch dimension for each feature. Layer normalisation normalises across the feature dimension for each example, making it independent of batch size. Layer normalisation is preferred in Transformers and RNNs, where variable batch sizes or sequential processing make batch normalisation impractical.