Back to Study Material Deep Learning

Vanishing gradient problem

The vanishing gradient problem occurs when gradients become extremely small as they are propagated backward through many layers during training. Because ea...

Vanishing gradient problem

Why Gradients Disappear in Deep Networks

The vanishing gradient problem occurs when gradients become extremely small as they are propagated backward through many layers during training. Because each layer multiplies the gradient by the derivative of its activation function and its weight matrix, chains of values smaller than 1 cause the gradient to shrink exponentially. By the time the signal reaches the earliest layers, the gradients are so tiny that those weights are barely updated, effectively preventing the network from learning long-range patterns.

The problem is especially severe with sigmoid and tanh activation functions. The sigmoid function has a maximum derivative of 0.25 at its centre, and this derivative approaches zero in the saturated regions of the function. Multiplying many such small values across dozens of layers produces a gradient close to zero in early layers.

Deep networks with many sigmoid layers trained on sequential data like text suffer from this problem severely. Early layers learn much more slowly than later layers, or may not learn at all, creating a mismatch in representation quality across the network.

Several solutions address vanishing gradients. ReLU activation functions have a derivative of 1 for positive inputs, preventing gradient shrinkage in active neurons. Residual connections (as in ResNets) add shortcut paths that carry the gradient directly to earlier layers, bypassing many multiplications. LSTM gated memory cells are designed to maintain gradient flow across long sequences in recurrent networks.

A concrete example: training a 20-layer network with sigmoid activations shows nearly zero parameter updates in the first five layers, while the last five layers train normally. Switching to ReLU immediately resolves the discrepancy and all layers train at similar speeds.

Exam tip: a common interview follow-up after explaining vanishing gradients is to ask about solutions. Always mention at least three: ReLU activations, residual connections, and careful weight initialisation (such as He or Xavier initialisation, which sets initial weights to a scale that maintains activation variance across layers). Listing all three with a brief explanation of each demonstrates comprehensive knowledge.