Back to Study Material Deep Learning

Regularization

Regularization refers to a collection of techniques that prevent a neural network from overfitting, which occurs when the model memorises the training data...

Regularization

Preventing Overfitting with Regularization

Regularization refers to a collection of techniques that prevent a neural network from overfitting, which occurs when the model memorises the training data so well that it fails to generalise to new examples. An overfit model has low training loss but high validation and test loss, indicating it has learned noise rather than true patterns.

The two most common weight-based regularization methods are L1 and L2. L2 regularization, also called weight decay, adds a penalty proportional to the sum of squared weights to the loss function. This discourages the network from assigning very large weights to any single feature, pushing the model toward smoother, more generalised solutions. L1 regularization adds a penalty proportional to the sum of absolute weight values, encouraging many weights to become exactly zero, effectively making the model sparse.

Dropout is another widely used technique where neurons are randomly set to zero during each training step with a specified probability. This prevents neurons from co-adapting too strongly and forces the network to learn redundant representations, making it more robust. At inference time, dropout is turned off and all neurons are active.

Data augmentation reduces overfitting for image models by artificially expanding the training set through transformations like flipping, rotation, cropping, and colour jitter. Each augmented version of an image teaches the network that these variations are irrelevant to the label, improving generalisation.

A concrete example: a model trained to classify skin lesions on a small medical dataset overfits badly without regularization. Adding L2 regularization and dropout reduces validation error significantly, producing a model that performs reliably on new patient images.

Exam tip: interviewers often ask when to apply each technique. Dropout is most effective in large fully connected layers. L2 regularization is a safe default for any layer with learned weights. Data augmentation is the first line of defence for image tasks with limited data. Knowing which tool to apply in which context shows practical experience.