Dropout
Dropout is a regularization technique where, during each training step, each neuron in a specified layer is independently set to zero with a given probabil...
Dropout as a Regularization Technique
Dropout is a regularization technique where, during each training step, each neuron in a specified layer is independently set to zero with a given probability p, typically between 0.2 and 0.5. This means different subsets of neurons are active for each training example, preventing neurons from relying too heavily on specific other neurons.
The motivation is ensemble learning. By randomly disabling neurons, dropout effectively trains a large number of different sub-networks simultaneously, all sharing the same weights. At inference time, dropout is disabled and all neurons are active. To compensate for the fact that all neurons now contribute, their outputs are scaled by the keep probability (1 minus p), maintaining the expected total activation.
Dropout is most effective in large fully connected layers, where co-adaptation between neurons is most problematic. It is less commonly applied to convolutional layers because spatial weight sharing already provides a form of regularization. Spatial dropout, which drops entire feature maps rather than individual neurons, is sometimes used for convolutional layers.
Dropout can be understood as adding noise to the representations learned by the network. This noise forces the model to learn more robust features that work across many different neural sub-configurations, improving generalisation to unseen data. The technique is especially valuable when the training set is small relative to the model's capacity.
A concrete example: a sentiment analysis model with three large fully connected layers applies dropout with p equal to 0.5 after each hidden layer. Without dropout, validation accuracy is 82 percent. With dropout, it rises to 87 percent as the model learns more generalisable sentiment features instead of memorising training examples.
Exam tip: interviewers sometimes ask why we scale activations at inference rather than training. Both approaches (inverted dropout at training time, scaling at inference time) are mathematically equivalent, but scaling at inference is more common because inference efficiency is critical in production. Inverted dropout keeps inference code simpler and faster, which is the standard implementation in PyTorch and TensorFlow.