Forward propagation
Forward propagation is the process by which input data passes through a neural network layer by layer, producing a final prediction. It is the first half o...
How Data Flows Through a Network
Forward propagation is the process by which input data passes through a neural network layer by layer, producing a final prediction. It is the first half of the training loop; the second half is backpropagation, which computes gradients. Understanding forward propagation is essential for understanding both inference and training.
At each layer, the computation is the same: multiply the input by the weight matrix, add the bias vector, and apply the activation function. For a layer with weight matrix W and bias vector b receiving input x, the output is f(Wx + b), where f is the activation function applied element-wise. This transformed output becomes the input for the next layer.
The final layer's output depends on the task. For binary classification, a single sigmoid neuron outputs a probability between 0 and 1. For multi-class classification, a softmax layer converts raw scores into a probability distribution. For regression, a linear output neuron produces an unbounded real number.
During inference (using a trained model to make predictions), only forward propagation is performed. The weights are fixed, and the network simply transforms the input into a prediction as efficiently as possible. Modern deep learning frameworks optimise this with GPU vectorisation, enabling millions of examples to be processed per second.
A concrete example: an MLP for handwritten digit recognition receives a flattened 784-element vector of pixel values. The first hidden layer transforms this into a 256-element vector of activations capturing low-level patterns. The second hidden layer maps these to 128 activations representing higher-level digit components. The output layer applies softmax to produce ten probabilities, one per digit class.
Exam tip: distinguish forward propagation from the full training step. Forward propagation is also called the forward pass. Training adds a loss computation, a backward pass (backpropagation), and a weight update. Inference uses only the forward pass. Candidates who conflate training and inference often lose marks on practical system design questions.