Attention mechanism
The attention mechanism allows a neural network to selectively focus on the most relevant parts of the input when producing each element of the output. Rat...
Attention: Letting the Model Focus Selectively
The attention mechanism allows a neural network to selectively focus on the most relevant parts of the input when producing each element of the output. Rather than compressing the entire input into a single fixed-length vector, attention computes a weighted sum of all input representations, with the weights indicating how much each input position contributes to the current output position.
In the original sequence-to-sequence attention proposed by Bahdanau and colleagues, a decoder at each output step computes an alignment score between its current hidden state and every encoder hidden state. These scores are passed through a softmax to produce attention weights that sum to 1. The weighted average of encoder hidden states forms the context vector used to produce the next output word, allowing the decoder to attend to the most relevant input words at each step.
Self-attention, as used in Transformers, computes attention within a single sequence. Each position computes queries, keys, and values from its own representation. Attention weights are computed as the dot product of a query with all keys, scaled and softmaxed, and the output is the weighted sum of values. This allows each position to directly gather information from any other position in the sequence in a single layer.
Multi-head attention runs multiple self-attention operations in parallel, each with different learned projections. The results are concatenated and linearly projected to the output dimension. This allows the model to attend to information from different representation subspaces simultaneously, capturing different types of relationships.
A concrete example: in machine translation, when translating the French word chat to cat, the attention mechanism assigns high weight to the French word and low weight to articles and prepositions, telling the decoder precisely which input word to use for the current output.
Exam tip: when asked how attention differs from traditional sequence models, emphasise that attention eliminates the bottleneck of a fixed-size hidden state. Every output position can directly access any input position. This direct access is what enables Transformers to model long-range dependencies that RNNs struggle with, which is a key reason Transformers dominate modern NLP.