Back to Study Material Deep Learning

Transformers

The Transformer is a deep learning architecture introduced in the 2017 paper "Attention Is All You Need" that revolutionised natural language processing an...

Transformers

The Transformer Architecture

The Transformer is a deep learning architecture introduced in the 2017 paper "Attention Is All You Need" that revolutionised natural language processing and later computer vision, audio, and biology. It processes sequences using a mechanism called self-attention rather than recurrence, enabling massive parallelisation during training and far better handling of long-range dependencies.

Self-attention allows every position in a sequence to directly attend to every other position in one step. For each word, the model computes a weighted sum of all other word representations, where the weights reflect how relevant each word is to understanding the current one. This is done simultaneously for all positions in parallel, unlike RNNs which process one step at a time.

A Transformer consists of an encoder and a decoder, each made of stacked layers. Each layer contains a multi-head self-attention sub-layer and a feed-forward sub-layer, with residual connections and layer normalisation around each. Multi-head attention runs several attention mechanisms in parallel and concatenates their outputs, allowing the model to attend to information from different representational subspaces simultaneously.

Positional encodings are added to input embeddings to give the model information about token order, since self-attention itself is position-agnostic. These are fixed sinusoidal functions in the original paper but can also be learned.

A concrete example: GPT models are Transformer decoders trained on massive text corpora to predict the next word. BERT is a Transformer encoder trained to fill in masked words. Both achieve state-of-the-art results across dozens of language benchmarks.

Exam tip: interviewers often ask about the computational complexity of self-attention. It is quadratic in sequence length because every token attends to every other token. For a sequence of length n, attention requires computing an n by n matrix. This is the main motivation behind efficient attention variants like sparse attention and linear attention, which are important topics for follow-up discussion.