Back to Study Material Deep Learning

Loss function

A loss function, also called a cost function or objective function, measures how wrong a neural network's predictions are compared to the true labels. The ...

Loss function

Measuring Error with Loss Functions

A loss function, also called a cost function or objective function, measures how wrong a neural network's predictions are compared to the true labels. The training process aims to minimise this value by adjusting the network's weights. Choosing the right loss function for the task is critical to getting good results.

For regression tasks, where the goal is to predict a continuous number, mean squared error (MSE) is the standard choice. MSE computes the average of the squared differences between predictions and actual values. Squaring the differences penalises large errors heavily and ensures the loss is always non-negative.

For binary classification, binary cross-entropy is used. It compares the predicted probability with the true binary label (0 or 1) using a logarithmic formula. If the model confidently predicts the correct class, the loss is nearly zero. If it confidently predicts the wrong class, the loss is very large, strongly pushing the weights to correct the mistake.

For multi-class classification, categorical cross-entropy extends this idea to multiple classes. The model produces a probability distribution over all classes via softmax, and categorical cross-entropy measures how far that distribution is from the true one-hot encoded label.

A concrete example: a model predicting house prices uses MSE. If it predicts 300,000 dollars when the true value is 350,000, the squared error is 2,500,000,000. Averaging this across all training examples gives the current loss. The optimizer then adjusts weights to reduce this figure over many iterations.

Exam tip: a common interview question is why we use cross-entropy instead of MSE for classification. The answer relates to gradient behaviour. MSE applied to sigmoid outputs produces very flat gradients when the model is confidently wrong, slowing learning significantly. Cross-entropy gradients remain steep in that situation, enabling faster and more stable convergence.