Back to Study Material Deep Learning

Transformer decoder

The transformer decoder generates output tokens one at a time in left-to-right order. Its self-attention mechanism uses a causal mask that prevents each po...

Transformer decoder

Causal Self-Attention and Autoregressive Decoding

The transformer decoder generates output tokens one at a time in left-to-right order. Its self-attention mechanism uses a causal mask that prevents each position from attending to future positions, ensuring that the prediction of each token depends only on the tokens already generated. This constraint is necessary for autoregressive generation but also allows the decoder to be trained efficiently: the entire target sequence is fed in at once with the mask applied, computing all position predictions in parallel rather than sequentially.

Consider a translation decoder generating the French translation of an English sentence. When predicting the third output word, the decoder is allowed to attend to its first and second output words but not to the fourth or later words, mimicking the condition under which it will operate at inference time.

Exam tip: The distinction between encoder self-attention and decoder self-attention is the causal mask. Encoder attention is fully bidirectional; decoder attention is unidirectional. In an encoder-decoder model, the decoder also contains a cross-attention sub-layer where the decoder's queries attend to the encoder's key and value representations, allowing the decoder to condition its output on the full source sequence.

Cross-Attention in Encoder-Decoder Models

In an encoder-decoder transformer, each decoder layer contains three sub-layers rather than two: causal self-attention, cross-attention, and a feed-forward network. The cross-attention layer allows each decoder position to attend to all encoder output positions. Queries come from the decoder's current representation, while keys and values come from the encoder's final layer output. This mechanism is the primary pathway through which the decoder incorporates information from the source sequence.

The cross-attention weights in a translation model often exhibit a roughly diagonal pattern, with each target word attending most strongly to its aligned source word, though this alignment is soft and flexible rather than hard as in older statistical translation models. Visualizing these attention patterns provides interpretable insight into the model's alignment behavior.

Decoder-only models used for language generation, such as the GPT family, omit the encoder and cross-attention entirely. All contextual information is provided through the causal self-attention mechanism over the prompt tokens, simplifying the architecture while retaining strong generation capability.

KV Cache and Inference Efficiency

During autoregressive inference, the decoder generates one token at a time. At each step, the self-attention mechanism needs the key and value vectors for all previously generated tokens to compute the current attention output. Recomputing these vectors from scratch at every step would be wasteful because the earlier tokens have not changed. The key-value cache, or KV cache, stores the key and value tensors from previous steps and reuses them, so only the new token's keys and values need to be computed at each step.

The KV cache significantly reduces redundant computation during generation, but it consumes memory proportional to the sequence length, the number of layers, and the number of attention heads. For long contexts or large batches, KV cache memory can become a bottleneck. Techniques such as grouped query attention reduce cache size by sharing key and value heads across multiple query heads, and sliding window attention limits cache size by only retaining a fixed-length recent context window.

Understanding the KV cache is a practical requirement for anyone working on language model inference infrastructure, making it a common interview topic in roles involving model deployment or serving system design.