Back to Study Material Deep Learning

Language model

A language model is a probability distribution over sequences of tokens. Given any sequence of text, a language model can assign a probability to that sequ...

Language model

What a Language Model Is

A language model is a probability distribution over sequences of tokens. Given any sequence of text, a language model can assign a probability to that sequence, indicating how likely it is to appear in natural language. Equivalently, a language model can generate text by repeatedly sampling the next token given the current sequence, extending the sequence one step at a time.

Early language models used n-gram statistics, estimating the probability of the next word from the preceding few words. These models were simple and fast but suffered from sparsity: most n-gram combinations never appear in training data, requiring smoothing techniques to assign nonzero probabilities. Neural language models replaced n-gram tables with neural networks that generalize far better across unseen contexts by learning dense vector representations of words and sentences.

Exam tip: When an exam question asks what a language model predicts, the precise answer is the probability distribution over the next token given the preceding context, not merely the most likely next word. Framing the answer in terms of probability distributions rather than single predictions shows technical precision.

Scale, Emergent Abilities, and the Scaling Law

Empirical research has found that language model performance on benchmarks improves predictably as model size and training data volume increase, following power-law scaling relationships. Plotting test loss against model parameters or training tokens on a log-log scale produces an approximately straight line, which allows researchers to predict the performance of larger models from experiments with smaller ones.

At sufficient scale, language models exhibit emergent abilities: capabilities that were not present at smaller scales and that appear somewhat suddenly as a threshold is crossed. Examples include multi-step arithmetic, language translation without task-specific fine-tuning, and analogical reasoning. These emergent capabilities are not explicitly trained for but arise from the model's broad exposure to human language and knowledge.

Understanding scaling laws is practically useful for planning training runs, because they allow teams to allocate compute between model size and training tokens optimally. The Chinchilla scaling analysis found that previous large models were substantially undertrained relative to their parameter count and that training a smaller model on more data often outperforms training a larger model on fewer tokens at the same compute budget.

Perplexity as an Evaluation Metric

Perplexity is the standard intrinsic metric for evaluating language models. It measures how surprised the model is, on average, by the tokens in a held-out test set. Formally, perplexity is the exponent of the average negative log probability per token, which equals two raised to the average cross-entropy loss per token when using log base two.

A model with lower perplexity is more confident about the correct next token on average. On a well-curated English text benchmark, early neural language models might achieve perplexity around 100, while large modern language models achieve perplexity below 10 on standard benchmarks. A perplexity of one would mean the model assigns probability one to every correct next token, which would require perfect prediction of the test text.

Perplexity does not directly measure the usefulness of a model for downstream tasks like question answering or summarization, and models with similar perplexity can differ substantially in practical capabilities. An interview answer that acknowledges both the utility of perplexity as a standardized comparable metric and its limitations as a proxy for end-task performance shows balanced technical judgment.