Optimizer
An optimiser is the algorithm that updates a neural network's weights based on the gradients computed by backpropagation. While all optimisers aim to minim...
Optimizers: From SGD to Adam
An optimiser is the algorithm that updates a neural network's weights based on the gradients computed by backpropagation. While all optimisers aim to minimise the loss function, they differ in how they use gradient information, how they adapt to different parameters, and how efficiently they converge.
Stochastic gradient descent (SGD) is the most basic optimiser. It updates each weight by subtracting the gradient multiplied by the learning rate. Pure SGD is simple and often generalises well, but it applies the same learning rate to every parameter and does not adapt over time, requiring careful tuning and learning rate schedules.
Momentum SGD improves on plain SGD by accumulating a velocity term. Instead of just moving in the direction of the current gradient, the update also carries momentum from previous gradients, smoothing the path toward the minimum and accelerating convergence in consistent directions while dampening oscillations in inconsistent ones.
Adam (Adaptive Moment Estimation) is the most popular optimiser in modern deep learning. It maintains a running average of both the first moment (mean of gradients) and the second moment (mean of squared gradients) for each parameter. By dividing the gradient by the square root of the second moment, it scales the effective learning rate individually for each parameter. Parameters that receive infrequent large gradients get smaller updates; those with small, consistent gradients get larger updates.
A concrete example: training a natural language processing model with Adam typically converges in far fewer epochs than plain SGD because Adam adapts the learning rate to each weight, handling sparse gradients from embedding layers far better.
Exam tip: a frequent interview question is whether Adam always outperforms SGD. The honest answer is no. For image classification with large datasets, SGD with momentum and a well-tuned schedule often achieves better final accuracy than Adam. Adam is preferred for faster convergence and robustness when hyperparameter tuning time is limited, but SGD can deliver superior generalisation given enough training budget.