Encoder decoder model
The encoder-decoder architecture is a neural network design pattern where one sub-network (the encoder) compresses the input into a compact representation,...
The Encoder Decoder Architecture
The encoder-decoder architecture is a neural network design pattern where one sub-network (the encoder) compresses the input into a compact representation, and another sub-network (the decoder) generates an output from that representation. This structure is suited to tasks where input and output have different forms or lengths, such as machine translation, image captioning, and text summarisation.
The encoder reads the input sequence step by step (in RNN-based models) or all at once (in Transformers) and produces a rich representation encoding the full input's meaning and structure. In early models, this was a single fixed-length vector: the final hidden state of the encoder. This bottleneck limited performance on long sequences because a single vector cannot faithfully represent hundreds of words.
Attention mechanisms extended the encoder-decoder framework by giving the decoder dynamic access to all encoder outputs rather than just the final state. At each decoding step, the decoder attends to different parts of the encoder output, allowing it to focus on the most relevant input information for the word it is currently generating.
In Transformer-based models, the encoder produces a sequence of contextualised representations and the decoder attends to all of them simultaneously using cross-attention. This is the architecture behind powerful translation and summarisation systems like T5 and BART.
A concrete example: an image captioning model uses a CNN encoder that converts an image into a compact feature vector or feature map. The decoder is an RNN or Transformer that generates a description word by word, attending to different spatial regions of the image at each step. When generating the word cat, the attention focuses on the region of the image containing the cat.
Exam tip: distinguish the encoder-decoder architecture from a plain sequence-to-sequence model. Seq2seq is the general task (input sequence to output sequence). Encoder-decoder is one family of architectures for solving it. Modern seq2seq models are almost exclusively Transformer-based encoder-decoders. Knowing this distinction and being able to name examples like T5 or BART positions you well for system design questions.