Multi-head attention
Multi-head attention extends self-attention by running several attention mechanisms in parallel, each with its own learned query, key, and value projection...
Why Multiple Heads Are Better Than One
Multi-head attention extends self-attention by running several attention mechanisms in parallel, each with its own learned query, key, and value projection matrices. Each parallel mechanism is called a head. The outputs from all heads are concatenated and linearly projected to produce the final output. Because each head uses different projections, each one can attend to different types of relationships simultaneously.
In a translation task, one head might specialize in syntactic agreement between subject and verb while another tracks semantic relationships between nouns and their modifiers. A single attention head would need to represent all of these relationships in a single weight matrix, but multiple heads can each focus on different patterns without interference.
Exam tip: The total computation of multi-head attention with h heads is not h times the cost of single-head attention if the per-head dimension is divided accordingly. Standard practice reduces the dimension of each head to the model dimension divided by the number of heads, keeping the total parameter count and computation roughly constant relative to a single full-dimensional head. Be able to explain this design choice.
Head Dimension, Projection Matrices, and Concatenation
In the original Transformer paper with a model dimension of 512 and eight heads, each head operates on vectors of dimension 64. Each head has three projection matrices of shape 512 by 64 for queries, keys, and values, and the output of each head is a matrix of shape sequence length by 64. The outputs of all eight heads are concatenated to form a matrix of shape sequence length by 512, which is then multiplied by an output projection matrix of shape 512 by 512 to produce the final multi-head attention output.
The output projection provides additional representational flexibility by allowing the model to mix information across heads after the fact, rather than simply treating the concatenated output as a fixed aggregation. Research has shown that this output projection contributes meaningfully to model quality.
When drawing the architecture diagram for multi-head attention, being precise about the shapes of all tensors at each step is a practical skill. Interviews at companies that build transformer-based systems frequently include questions that require tracing tensor shapes through the architecture.
Head Specialization and Pruning Research
Empirical studies of trained transformer models have found that individual attention heads often develop consistent behavioral specializations. Some heads consistently attend to syntactic dependencies, others track positional relationships, and some appear to perform retrieval of related facts. This specialization emerges from training on raw text and is not explicitly designed into the architecture.
Research on attention head pruning has found that many heads can be removed from a trained model with minimal loss in performance, suggesting that not all heads are equally important. Removing redundant heads reduces model size and inference cost. The fact that pruning works selectively, with some heads being critical and others nearly irrelevant, supports the view that individual heads carry distinct information.
An interview question about the purpose of multiple attention heads benefits from mentioning both the design motivation, allowing parallel attention to different relationship types, and the empirical observation that heads specialize during training. This combination of theoretical motivation and experimental evidence strengthens the answer.