RNN and LSTM
Recurrent Neural Networks (RNNs) are designed to process sequential data where the order of elements matters, such as text, speech, and time-series data. U...
Recurrent Networks and Long Short-Term Memory
Recurrent Neural Networks (RNNs) are designed to process sequential data where the order of elements matters, such as text, speech, and time-series data. Unlike feedforward networks that treat each input independently, an RNN maintains a hidden state that acts as a memory, updated at each time step by combining the current input with information from previous steps.
The hidden state at each time step is computed from the current input and the previous hidden state using shared weights. This weight sharing across time steps means the network applies the same transformation at every position in the sequence, which is efficient and allows processing sequences of variable length.
However, standard RNNs struggle with long sequences. During backpropagation through many time steps, gradients are multiplied together repeatedly. If weights are slightly below 1, these multiplications cause gradients to shrink exponentially, making it impossible to learn dependencies between elements far apart in a sequence. This is the vanishing gradient problem.
Long Short-Term Memory (LSTM) networks solve this with a gated architecture. An LSTM cell maintains both a hidden state and a cell state. Three learned gates, the forget gate, input gate, and output gate, control what information is erased, added, and read from the cell state at each time step. The cell state can carry relevant information across hundreds of time steps without significant degradation.
A concrete example: machine translation with an RNN-based encoder reads a French sentence word by word, building up a hidden state that summarises the entire sentence. The decoder then generates an English translation step by step, conditioned on that summary state.
Exam tip: interviewers frequently contrast RNNs with LSTMs. The essential answer is that standard RNNs cannot learn long-range dependencies due to vanishing gradients, while LSTMs address this with gated memory cells. Mention that Transformers have largely replaced LSTMs for language tasks, but LSTMs remain relevant for time-series and streaming applications.