Gradient descent
Gradient descent is the optimisation algorithm that minimises the loss function by iteratively updating the network's weights in the direction that reduces...
How Gradient Descent Optimises a Network
Gradient descent is the optimisation algorithm that minimises the loss function by iteratively updating the network's weights in the direction that reduces the error. The word "gradient" refers to the vector of partial derivatives of the loss with respect to each weight, which points in the direction of steepest increase. Moving in the opposite direction decreases the loss.
The update rule is simple: each weight is reduced by the gradient of the loss with respect to that weight, multiplied by a small constant called the learning rate. A high learning rate causes large steps, which can overshoot the minimum and cause training to diverge. A low learning rate causes slow, steady progress but may get stuck in local minima or take too long to converge.
There are three main variants. Batch gradient descent computes the gradient using the entire training dataset before taking a step. This is accurate but slow for large datasets. Stochastic gradient descent (SGD) updates weights after every single training example, making it fast but noisy. Mini-batch gradient descent, the most widely used approach, computes the gradient on a small random subset of the data called a mini-batch, balancing speed and accuracy.
A concrete example: imagine the loss function as a hilly landscape and the current weight values as a ball placed somewhere on that landscape. Gradient descent rolls the ball downhill by nudging it in the direction the slope descends. With each training step the ball moves closer to a valley, representing a minimum loss.
Exam tip: interviewers often ask about the difference between the three variants. The core trade-off is computational cost versus gradient accuracy. Mini-batch gradient descent is the practical standard because it is fast, takes advantage of GPU parallelism, and produces gradient estimates accurate enough for reliable convergence. Knowing this trade-off and articulating it clearly is a strong answer.