Back to Study Material Deep Learning

Tokenization

Neural networks operate on numerical tensors, not raw text. Tokenization is the process of converting a string of text into a sequence of integer indices t...

Tokenization

Why Tokenization Is Necessary

Neural networks operate on numerical tensors, not raw text. Tokenization is the process of converting a string of text into a sequence of integer indices that the model can process. The mapping is defined by a vocabulary: a fixed list of text units, each assigned a unique integer. Every input string is segmented into units from this vocabulary, and the resulting sequence of integers is fed to the embedding layer.

Character-level tokenization uses individual characters as units, producing very long sequences but a compact vocabulary. Word-level tokenization uses whole words, resulting in shorter sequences but a vocabulary that cannot represent rare or unseen words. Subword tokenization methods such as Byte Pair Encoding and WordPiece balance these trade-offs by using common words as single tokens while splitting rare words into smaller subunits that still appear in the vocabulary.

Exam tip: Byte Pair Encoding starts with a character vocabulary and iteratively merges the most frequent adjacent pair of units into a new token until the desired vocabulary size is reached. Be able to describe this algorithm in a few sentences and explain why it handles rare and compound words better than word-level tokenization.

Subword Tokenization in Practice

The GPT family of models uses a variant of BPE applied at the byte level, ensuring that any input text can be represented without encountering unknown tokens, because every possible byte value is in the base vocabulary. BERT uses WordPiece, which is similar to BPE but selects merges based on likelihood rather than raw frequency. SentencePiece is a language-independent framework that implements both BPE and unigram language model tokenization and operates directly on raw Unicode text without requiring pre-tokenization into words.

A concrete example: the word unhappiness might be tokenized as un, happy, and ness by a subword tokenizer, allowing the model to reuse these subunits across related words like unhappy, happiness, and happily. This sharing of subword representations helps the model generalize across morphologically related forms.

The vocabulary size is a critical hyperparameter. A vocabulary of 32,000 to 100,000 tokens is typical for large language models. Larger vocabularies reduce sequence length, which reduces computation, but require a larger embedding matrix and may produce less frequent tokens that are harder to learn well.

Tokenization Artifacts and Their Consequences

Tokenization choices have non-obvious effects on model behavior. Arithmetic is harder for language models partly because numbers are tokenized inconsistently: the number 123 might be a single token while 124 is split into 12 and 4, making it difficult for the model to learn systematic numerical relationships. Similarly, words in languages with rich morphology may be split into many subword tokens, resulting in much longer sequences for the same semantic content compared to morphologically simpler languages and effectively reducing the available context window.

Whitespace handling differs between tokenizers and can cause surprising behavior. Some tokenizers include the leading space of a word as part of the token, meaning the token for the appears as a distinct vocabulary entry from the same letters without a preceding space. This means that prompts with unusual spacing may tokenize differently than expected.

Awareness of tokenization artifacts is useful in interviews when discussing why language models struggle with certain tasks. Connecting the failure mode, such as poor performance on character-level string manipulation, to the underlying cause, namely that the model never sees individual characters during training, demonstrates a concrete understanding of the full model pipeline from raw text to prediction.