Back to Study Material Deep Learning

Transformer encoder

The transformer encoder processes an input sequence and produces a context-sensitive representation for every token. Unlike recurrent networks, which proce...

Transformer encoder

The Role and Structure of the Encoder

The transformer encoder processes an input sequence and produces a context-sensitive representation for every token. Unlike recurrent networks, which process tokens sequentially, the encoder processes all tokens in parallel. Each encoder layer applies self-attention to allow every token to incorporate information from all other tokens, followed by a position-wise feed-forward network that applies a small multi-layer perceptron independently to each token position.

In the original encoder-decoder transformer designed for machine translation, the encoder reads the source language sentence and produces representations that capture both local syntax and long-range relationships. These representations are then used by the decoder to generate the translated sentence in the target language.

Exam tip: The encoder uses bidirectional self-attention, meaning every token can attend to every other token in both the forward and backward directions. This is the key difference from a causal decoder, which only allows each token to attend to previous tokens. BERT is a well-known encoder-only model that exploits bidirectionality for tasks like sentence classification and named entity recognition.

Layer Normalization and Residual Connections

Each sub-layer within a transformer encoder block, the self-attention mechanism and the feed-forward network, is wrapped with a residual connection and a layer normalization step. The residual connection adds the input of the sub-layer directly to its output before normalization, following the design popularized by ResNet. This addition creates a direct gradient path that bypasses the sub-layer, making it easier to train deep stacks of layers because gradients flow back through the residual path without vanishing.

Layer normalization standardizes the activations within each training example across the feature dimension, unlike batch normalization which operates across the batch. This makes behavior more consistent when the batch size varies and is particularly appropriate for sequence models where different positions may have very different activation magnitudes.

The combination of residual connections and layer normalization is so fundamental to transformer training stability that nearly every modern large language model uses it, often with minor variations such as pre-norm placement, where normalization is applied before the sub-layer rather than after, which is now the more common convention.

Feed-Forward Sub-Layer and Encoder Depth

The position-wise feed-forward network within each encoder layer consists of two linear transformations with a nonlinear activation between them, typically a ReLU or GELU function. The intermediate dimension is usually four times the model dimension, so a model with dimension 512 uses an intermediate size of 2048. This expansion and contraction pattern allows the layer to perform complex per-token transformations while returning to the original dimension for the next layer's input.

Research interpreting the role of feed-forward layers in language models has found that they function as associative memories that store and retrieve factual knowledge. When a model answers a question about a specific fact, the relevant information is often retrieved by a specific feed-forward layer rather than encoded in the attention mechanism.

Encoder depth, meaning the number of stacked layers, is one of the most important scaling dimensions. BERT-base uses twelve layers, while large research models use hundreds. Each additional layer allows the model to compose representations from the previous layer, enabling increasingly abstract and task-relevant features to emerge.