Epoch and batch size
In deep learning training, an epoch is one complete pass through the entire training dataset. A batch is a subset of the training data used to compute one ...
Training Dynamics: Epochs and Batches
In deep learning training, an epoch is one complete pass through the entire training dataset. A batch is a subset of the training data used to compute one gradient update. These two concepts together control how the model learns from data and significantly influence training speed, stability, and generalisation.
When the training dataset is large, processing all examples before a single weight update would be extremely slow and memory-intensive. Instead, the data is divided into mini-batches. After each mini-batch, gradients are computed and the weights are updated. One epoch consists of processing all mini-batches in sequence until every training example has been seen once.
Batch size affects both training dynamics and hardware utilisation. Small batches (16 to 64 examples) produce noisy gradient estimates, meaning the parameter updates are less accurate but more varied. This noise can help the model escape sharp local minima and often leads to better generalisation. Large batches (512 to 4096) produce smoother gradients, allowing faster and more stable convergence per update, but may converge to sharper minima that generalise less well.
The number of epochs is a hyperparameter chosen through experimentation. Too few epochs means the model is undertrained. Too many causes overfitting as the model begins memorising training data. Early stopping monitors validation loss and halts training when it stops improving, automatically finding a good epoch count.
A concrete example: a dataset of 60,000 images with a batch size of 200 produces 300 mini-batches per epoch. Running 50 epochs means 15,000 total weight updates. Plotting training and validation loss across epochs reveals whether the model is still learning, has converged, or has started overfitting.
Exam tip: interviewers often ask about the relationship between batch size and learning rate. Larger batches produce more accurate gradients, so a proportionally larger learning rate is often used (linear scaling rule). However, this must be tuned carefully; simply doubling the batch size and learning rate simultaneously does not always work well in practice without a warmup schedule.