Back to Study Material Deep Learning

Exploding gradient problem

The exploding gradient problem is the counterpart to vanishing gradients. It occurs when gradients grow exponentially as they are propagated backward throu...

Exploding gradient problem

When Gradients Grow Too Large

The exploding gradient problem is the counterpart to vanishing gradients. It occurs when gradients grow exponentially as they are propagated backward through many layers. When weight matrices have eigenvalues greater than 1, repeated multiplication causes gradients to grow without bound, leading to numerically unstable updates where weights jump to very large values or become NaN (not a number), completely destabilising training.

The problem is particularly common in recurrent neural networks processing long sequences, where the same weight matrix is applied at every time step. Over many steps, small instabilities compound. It is also seen in very deep feedforward networks with poor weight initialisation or excessively high learning rates.

The clearest symptom of exploding gradients is a loss that suddenly jumps to an extremely large value or becomes NaN. Inspecting the norm of the gradient vector reveals values in the thousands or millions instead of single digits.

Gradient clipping is the standard solution. Before applying the parameter update, the gradient vector's norm is computed. If it exceeds a threshold (commonly 1.0 to 5.0), the gradient is rescaled so its norm equals the threshold. This preserves the direction of the gradient while bounding its magnitude, preventing catastrophic updates while still allowing effective learning.

Proper weight initialisation, smaller learning rates, batch normalisation, and gradient clipping together form a robust defence against exploding gradients. Modern optimisers like Adam are less susceptible than plain SGD because adaptive learning rates naturally limit the magnitude of parameter updates.

A concrete example: an LSTM trained on very long text sequences without gradient clipping diverges after 10 epochs with loss values becoming NaN. Adding gradient clipping with a max norm of 5.0 stabilises training, and the model converges normally to a low loss within 30 epochs.

Exam tip: interviewers expect you to contrast vanishing and exploding gradients clearly. Vanishing gradients prevent learning in early layers; exploding gradients cause numerical instability. Vanishing is addressed with ReLU, residual connections, and LSTM gates. Exploding is addressed with gradient clipping and proper initialisation. Being able to name both problems, their causes, and their solutions in one fluent answer is a strong indicator of solid deep learning fundamentals.