Fine tuning
Fine tuning is the process of continuing to train a pretrained model on a new dataset with a small learning rate, allowing the pretrained weights to adapt ...
Fine Tuning Pretrained Models for Specific Tasks
Fine tuning is the process of continuing to train a pretrained model on a new dataset with a small learning rate, allowing the pretrained weights to adapt to the specific characteristics of the target task while retaining the general knowledge encoded from the original large-scale training. It is a more thorough form of transfer learning compared to only training a new classification head.
When fine tuning, all or most of the network's weights are updated, not just the final layer. Because the pretrained weights are already good starting points, only small adjustments are needed. Using a very small learning rate (often ten to one hundred times smaller than the rate used for training from scratch) ensures the pretrained representations are not destroyed but are instead gently steered toward the new domain.
A common strategy is to use layer-wise learning rate decay, where earlier layers of the network (which encode more general features) are updated with smaller learning rates than later layers (which encode more task-specific features). This recognises that general features like edges and textures need very little adjustment, while the final layers encoding high-level semantics need more significant changes.
Gradual unfreezing is another effective technique: first train only the new head for a few epochs, then unfreeze the last few layers and train with a lower rate, and finally unfreeze all layers and train at the lowest rate. This avoids catastrophic forgetting, where aggressive fine tuning destroys the pretrained representations.
A concrete example: a BERT language model pretrained on a large text corpus is fine-tuned for customer review sentiment classification by training all layers on 10,000 labelled reviews with a learning rate of 2 times 10 to the power of negative 5. The result outperforms a purpose-built sentiment model trained from scratch on the same data.
Exam tip: distinguish fine tuning from feature extraction. Feature extraction freezes all pretrained layers and only trains the new head. Fine tuning updates all layers. The choice depends on dataset size: small datasets call for feature extraction to avoid overfitting; larger datasets benefit from full fine tuning.