Back to Study Material Deep Learning

Model compression

Large neural networks achieve strong accuracy but their deployment is constrained by memory, latency, and energy requirements. A model with billions of par...

Model compression

Why Model Compression Matters

Large neural networks achieve strong accuracy but their deployment is constrained by memory, latency, and energy requirements. A model with billions of parameters cannot run on a smartphone or an embedded sensor without compression. Even in data center settings, reducing model size and inference cost translates directly into cost savings and lower latency, enabling new applications or serving more users with the same hardware budget.

Model compression is the set of techniques that reduce the resource requirements of a trained model, ideally while preserving most of its accuracy. The most common categories are pruning, quantization, knowledge distillation, and low-rank decomposition. These techniques are often combined: a model might be distilled into a smaller architecture, quantized to 8-bit integers, and then pruned to remove remaining redundant weights.

Exam tip: When discussing model compression in an interview, connect each technique to the specific resource it reduces. Pruning reduces parameter count and potentially FLOPs. Quantization reduces memory footprint and can accelerate inference on hardware with integer arithmetic units. Distillation reduces model depth or width. Distinguishing which resource is the bottleneck in a given deployment scenario helps justify the choice of technique.

Pruning: Structured and Unstructured

Pruning removes weights or entire structures from a neural network. Unstructured pruning removes individual weights based on their magnitude, setting small weights to zero. This can achieve high sparsity ratios but produces irregular sparse matrices that do not accelerate computation on standard hardware unless dedicated sparse tensor cores or specialized inference libraries are used.

Structured pruning removes larger units such as entire convolutional filters, attention heads, or layers. A network with several attention heads removed is still dense in its remaining structure and runs faster on standard hardware without specialized sparse arithmetic support. Structured pruning is therefore more practically useful for deployment on conventional hardware, even though it typically allows less aggressive compression than unstructured pruning at the same accuracy level.

The standard pruning workflow is to train a full model, prune it to a target sparsity, and then fine-tune the pruned model to recover accuracy lost by removing weights. The lottery ticket hypothesis suggests that the sparse subnetworks found by pruning could have been trained to similar performance from the start if their connectivity pattern were known in advance.

Quantization Methods and Trade-offs

Quantization reduces the numerical precision of model weights and activations. Post-training quantization applies calibration data to determine appropriate scaling factors after training is complete, typically converting 32-bit or 16-bit floating point to 8-bit integers without any additional training. This is fast and requires no labeled data beyond a small calibration set, making it a common first step in deployment optimization.

Quantization-aware training simulates quantization arithmetic during training, allowing the model to adapt its weights to minimize accuracy loss under the precision reduction. This produces better accuracy than post-training quantization at the cost of rerunning the training or fine-tuning phase.

Very low bit quantization, such as 4-bit or even binary weights, is increasingly used for extremely constrained deployments. Methods like GPTQ and AWQ apply quantization specifically to large language model weights with sophisticated calibration to minimize accuracy degradation at 4-bit precision. The trade-off between bit width, accuracy, and inference speed is hardware-dependent, and profiling on the actual target device is the only reliable way to determine the optimal compression configuration.