Activation functions
Activation functions introduce non-linearity into a neural network. Without them, stacking many layers would be equivalent to a single linear transformatio...
Why Activation Functions Matter
Activation functions introduce non-linearity into a neural network. Without them, stacking many layers would be equivalent to a single linear transformation, no matter how deep the network. Non-linearity is what allows neural networks to approximate complex, real-world functions.
The sigmoid function outputs values between 0 and 1, making it useful for binary classification output layers. Its formula is 1 divided by (1 plus e to the power of negative z). However, sigmoid suffers from the vanishing gradient problem in deep networks because its gradient becomes very small for large positive or negative inputs, slowing learning in earlier layers.
The ReLU (Rectified Linear Unit) function solves this by outputting zero for negative inputs and the input itself for positive inputs. Its gradient is either 0 or 1, making it computationally cheap and resistant to vanishing gradients in most of the network. ReLU and its variants like Leaky ReLU and ELU are the default choice for hidden layers in modern deep learning.
The softmax function is used in output layers for multi-class classification. It converts a vector of raw scores into a probability distribution, ensuring all outputs sum to 1. A ten-class image classifier uses softmax as the final activation so that each output neuron represents the probability of one class.
A concrete example: in a fruit recognition model, a hidden layer neuron with ReLU will output a positive score when it detects a curved edge (relevant to apples and oranges) and output zero otherwise. This sparse activation is computationally efficient and helps the network represent information more cleanly.
Exam tip: interviewers frequently ask why we cannot use a linear activation function throughout. The answer is that composing linear functions always results in a linear function, regardless of depth. Non-linear activations like ReLU are what give deep networks their expressive power. Stating this clearly and concisely demonstrates a solid conceptual foundation.