Learning rate
The learning rate is a hyperparameter that controls how large a step the optimiser takes in the direction of the negative gradient during each weight updat...
Choosing and Tuning the Learning Rate
The learning rate is a hyperparameter that controls how large a step the optimiser takes in the direction of the negative gradient during each weight update. It is one of the most important and sensitive hyperparameters in deep learning training, with a strong influence on whether training converges at all and how quickly.
If the learning rate is too high, weight updates are large and the optimiser may overshoot the loss minimum, causing the loss to oscillate or even diverge. If it is too low, training progresses extremely slowly and may get stuck in local minima or saddle points, never reaching a good solution within a practical number of epochs.
Typical starting values are between 0.1 and 0.0001, depending on the architecture and optimiser. The Adam optimiser, which adapts the learning rate per parameter, is generally less sensitive to the initial learning rate choice than plain SGD.
Learning rate schedules gradually reduce the learning rate during training. Step decay reduces the rate by a fixed factor every few epochs. Cosine annealing smoothly decreases the rate following a cosine curve and can include warm restarts to escape local minima. The one-cycle policy, popular in practice, ramps the learning rate up and then down over the entire training run, achieving fast convergence and strong generalisation.
A concrete example: training a ResNet-50 image classifier starts with a learning rate of 0.1 using SGD. After 30 epochs the rate is multiplied by 0.1, and again at epoch 60. This schedule reliably reaches state-of-the-art accuracy on ImageNet.
Exam tip: the learning rate finder technique is worth mentioning in interviews. Run a few mini-batches of training while exponentially increasing the learning rate from a very small to a very large value. Plot loss versus learning rate and choose the value just before the loss starts rising. This systematic approach saves hours of trial-and-error hyperparameter search.