Self-attention
Self-attention is a mechanism that allows a neural network to compute a representation of each element in a sequence by considering how relevant every othe...
The Intuition Behind Self-Attention
Self-attention is a mechanism that allows a neural network to compute a representation of each element in a sequence by considering how relevant every other element in the same sequence is to it. For each position, the mechanism produces a weighted sum of value vectors from all positions, where the weights reflect pairwise relevance scores. This allows the model to incorporate long-range context directly, without the information having to pass through many sequential steps as in recurrent networks.
Consider the sentence the animal did not cross the street because it was too tired. Resolving what it refers to requires attending to animal rather than street. Self-attention can learn to assign high weight to animal when computing the representation of it, enabling correct coreference resolution regardless of the distance between the two tokens.
Exam tip: The three vectors involved in self-attention are query, key, and value, all derived from the same input sequence by linear projection. The attention weight between position i and position j is the softmax-normalized dot product of the query at i and the key at j. Memorize this formulation and be able to write the attention formula from scratch.
Scaled Dot-Product Attention Mechanics
The full computation of scaled dot-product attention proceeds as follows. Given an input matrix where each row is a token embedding, three weight matrices project the input into query, key, and value matrices of the same sequence length. The attention score matrix is computed by multiplying the query matrix by the transpose of the key matrix, producing a square matrix of raw scores. These scores are divided by the square root of the key dimension to prevent the dot products from becoming so large that softmax pushes most weight to a single position. Softmax is applied along the last dimension to produce normalized attention weights, and these weights are used to compute a weighted sum of the value vectors.
The scaling by the square root of the key dimension is a small but important detail. Without it, the dot products grow in magnitude as the dimension increases, pushing the softmax into a nearly one-hot regime where gradients become very small, slowing training.
Writing out the matrix multiplication steps and the shapes at each stage is a useful exercise for solidifying understanding of this mechanism before a technical interview.
Computational Complexity and Efficient Attention Variants
The primary limitation of standard self-attention is that computing the attention score matrix requires multiplying every pair of positions, producing a complexity of order N squared in sequence length N, both in computation and in memory for storing the scores. For short sequences this is fine, but at context lengths of tens of thousands of tokens the quadratic scaling becomes prohibitive.
Efficient attention variants address this in different ways. Sparse attention patterns restrict each token to attending to only a local neighborhood plus a few global positions. Linear attention approximates the softmax attention with a kernel function that allows the computation to be reformulated as a linear rather than quadratic operation. Flash Attention, a widely adopted implementation technique, does not change the mathematical result but reorders computation to minimize slow high-bandwidth memory accesses on GPU, achieving significant speed-ups in practice without approximation.
Understanding why standard attention is quadratic and being able to describe at least one approach to reducing this cost is a strong differentiator in interviews about transformer efficiency.