Sequence to sequence model
A sequence-to-sequence (seq2seq) model is a neural network that maps an input sequence to an output sequence, where the two sequences can differ in length ...
Mapping One Sequence to Another
A sequence-to-sequence (seq2seq) model is a neural network that maps an input sequence to an output sequence, where the two sequences can differ in length and content. This is the core problem in machine translation, speech recognition, text summarisation, code generation, and question answering, making seq2seq one of the most impactful architectures in modern deep learning.
Early seq2seq models used RNN-based encoder-decoder pairs. The encoder read the input word by word and produced a context vector summarising the sequence. The decoder generated the output word by word, conditioned on this context. The main limitation was the fixed-size context vector, which struggled to represent long inputs.
The addition of attention solved the bottleneck by giving the decoder dynamic access to the full encoder output at each step. This improvement dramatically increased performance on long sequences, particularly in machine translation, where the relationship between distant words must be captured.
Transformer-based seq2seq models replaced RNNs with self-attention and cross-attention layers. The encoder processes the full input simultaneously, producing rich contextualised representations. The decoder generates the output left to right, attending to the encoder output and its own previous outputs at each step. This architecture powers modern large language models.
Beam search is a widely used decoding strategy for seq2seq models. Instead of greedily choosing the highest probability word at each step, beam search maintains k candidate sequences and expands each at every step, keeping only the top k sequences by cumulative probability. This produces more fluent and accurate outputs than greedy decoding.
A concrete example: Google Translate's Neural Machine Translation system is a seq2seq model. It encodes an input sentence in one language, and the decoder generates the translation token by token in the target language, guided by attention over the encoded input.
Exam tip: interviewers often ask about the exposure bias problem in seq2seq training. During training, the decoder receives true previous tokens (teacher forcing). During inference, it uses its own predictions. If early predictions are wrong, errors accumulate across the sequence. Scheduled sampling and reinforcement learning-based fine tuning are techniques used to narrow this train-test discrepancy.