Embedding layer
An embedding layer is a lookup table that maps each integer token index to a dense continuous vector of fixed dimension. When a token index is passed in, t...
Turning Discrete Tokens into Continuous Vectors
An embedding layer is a lookup table that maps each integer token index to a dense continuous vector of fixed dimension. When a token index is passed in, the embedding layer returns the corresponding row of a learned weight matrix. This converts the sparse, discrete representation of a token into a continuous vector that captures semantic and syntactic relationships learned during training.
For example, in a well-trained word embedding space, the vectors for king, queen, man, and woman are positioned such that the vector difference between king and man is approximately the same as the difference between queen and woman, encoding the gender relationship as a geometric transformation. This arithmetic structure emerges from learning to predict context words across millions of examples rather than from any explicit supervision signal.
Exam tip: The embedding layer is simply a weight matrix of shape vocabulary size by embedding dimension. Indexing into it with an integer is equivalent to a one-hot vector multiplied by the weight matrix. Understanding this equivalence helps explain why embedding layers and linear layers are related and why the output projection of a language model is sometimes tied to the embedding weights.
Embedding Dimensionality and Initialization
The embedding dimension is a hyperparameter that controls the richness of the token representations. Larger dimensions allow the model to encode more nuanced semantic information but increase the parameter count of the embedding matrix and the computational cost of subsequent layers. Typical values range from 128 dimensions for small task-specific models to 12,288 dimensions for large language models like GPT-4.
Initialization matters significantly for embedding quality. Random initialization sampled from a small normal or uniform distribution is the standard starting point. Initializing with pretrained embeddings from models like Word2Vec or GloVe can accelerate training for small datasets by providing a head start with already-meaningful representations, though for large-scale pretraining from scratch, random initialization works well because the model will see enough data to learn good embeddings regardless.
A practical consideration is that the embedding matrix for large vocabularies can be a significant fraction of the model's total parameter count. A vocabulary of 50,000 tokens with an embedding dimension of 1,024 requires 50 million parameters just for the embedding, comparable to the size of many mid-sized classification networks.
Positional Embeddings and Beyond
Token embeddings represent the identity of a token but not its position in the sequence. In a self-attention based model, all positions are treated symmetrically unless position information is added explicitly. Positional embeddings solve this by adding a position-dependent vector to each token embedding, encoding the sequence order in the same vector space.
The original Transformer paper used fixed sinusoidal functions of different frequencies to generate positional embeddings, producing a unique pattern for each position without any learned parameters. Learned positional embeddings, used in GPT and BERT, treat position indices exactly like token indices and look up their embeddings from a separate learned matrix, adapting position representations to the training data.
More recent architectures use rotary positional embeddings or ALiBi, which encode relative rather than absolute positions and generalize more gracefully to sequence lengths longer than those seen during training. An interview discussion about positional embeddings can demonstrate depth by explaining why absolute position encodings struggle at long contexts and how relative position methods address this limitation.