Back to Study Material Deep Learning

Knowledge distillation

Knowledge distillation is a model compression technique in which a small student model is trained to mimic the behavior of a larger teacher model. Rather t...

Knowledge distillation

The Teacher-Student Framework

Knowledge distillation is a model compression technique in which a small student model is trained to mimic the behavior of a larger teacher model. Rather than training the student on hard labels that assign probability one to the correct class and zero to all others, distillation uses the teacher's output probability distribution, called soft labels or soft targets, as the training signal. Soft targets contain richer information than hard labels because they reveal how the teacher distributes probability across incorrect classes, encoding similarity relationships between classes.

For example, a teacher classifying images of cats and dogs assigns near-zero but nonzero probability to dog when shown a cat. The soft label reflects that cats and dogs are both animals and share some visual features, while the hard label treats all incorrect classes identically. The student that learns from soft labels acquires this relational knowledge implicitly, often generalizing better than one trained from hard labels alone.

Exam tip: The temperature parameter in distillation controls how soft the teacher's probability distribution is. High temperature makes the distribution more uniform, amplifying the signal carried by near-zero probabilities. Low temperature makes it more peaked. A temperature of one recovers the standard softmax, while a temperature of 20 or higher makes all classes nearly equally probable. Understanding the role of temperature is a common interview question on distillation.

Feature Distillation and Intermediate Layers

Output distillation uses only the teacher's final layer predictions. Feature distillation extends this by training the student to match the teacher's intermediate layer activations as well. By aligning representations at multiple layers, the student is guided not only toward the correct output but also toward producing similar internal computations along the way, which tends to improve performance further.

FitNets was an early example of this approach, using hint-based training where intermediate student layer outputs are trained to match corresponding teacher layers through a learned regression layer. Later methods such as AT, which stands for attention transfer, align the spatial attention maps generated by each layer rather than the raw feature values, providing a more spatially compact and task-relevant distillation signal.

In practice, the choice of which layers to align is a hyperparameter that benefits from experimentation. Aligning too many intermediate layers can constrain the student's learning excessively, particularly when the student and teacher differ substantially in architecture, limiting the student to representations that fit a teacher designed around different capacity assumptions.

Self-Distillation and Applications to Language Models

Self-distillation applies the distillation framework without a separate teacher, using the model's own predictions from an earlier training epoch or a different dropout mask as the soft labels. This regularization effect often improves generalization by preventing the model from becoming overconfident on individual training examples. Born-again networks are a systematic exploration of this idea, showing that iteratively distilling a model into a copy of itself produces consistent accuracy improvements.

For large language models, distillation has produced notable results. DistilBERT is a distilled version of BERT that retains about 97 percent of BERT's performance on the GLUE benchmark with 40 percent fewer parameters and 60 percent of the inference speed. It was trained using a combination of output distillation on the masked language modeling objective and cosine-similarity alignment of hidden state representations.

Distillation is also used to transfer instruction-following capabilities from large commercial models to smaller open-source models. Training a small model on responses generated by a large teacher model allows the student to learn the teacher's output distribution directly, though this practice raises questions about licensing terms that require attention in production applications.