Fine-tuned language model
Fine-tuning adapts a pretrained language model to perform well on a specific task or in a specific domain by continuing training on a smaller, task-relevan...
The Purpose and Process of Fine-Tuning
Fine-tuning adapts a pretrained language model to perform well on a specific task or in a specific domain by continuing training on a smaller, task-relevant dataset. The pretrained weights encode broad language understanding, and fine-tuning adjusts them to match the distribution and requirements of the target task without losing the general capabilities learned during pretraining.
A medical language model is a concrete example. A general-purpose language model pretrained on internet text has some medical knowledge, but fine-tuning it on clinical notes, research papers, and drug interaction databases produces a model that answers medical questions more accurately and uses domain-appropriate terminology. The fine-tuning dataset is orders of magnitude smaller than the pretraining corpus, yet the improvement in domain-specific performance is substantial.
Exam tip: Full fine-tuning updates all model parameters, which requires storing a full set of gradients and optimizer states, roughly three times the memory of the model weights alone for Adam. Parameter-efficient fine-tuning methods such as LoRA address this by adding and training only a small number of new parameters while keeping the original weights frozen. Be prepared to explain LoRA's approach of low-rank weight updates.
Instruction Fine-Tuning and Alignment
Instruction fine-tuning trains a language model on examples of the form instruction, input, desired output, teaching the model to follow explicit task descriptions rather than relying on the user to engineer a specific prompt format. Models fine-tuned this way respond more reliably to natural language instructions, making them suitable for general-purpose assistant applications.
The FLAN model family demonstrated that fine-tuning on a large collection of diverse instruction-following tasks significantly improved zero-shot performance on unseen tasks, because the model learns the general skill of following instructions rather than memorizing any single task. Subsequent work by OpenAI and others added reinforcement learning from human feedback on top of instruction fine-tuning to further align model behavior with human preferences regarding helpfulness and safety.
Understanding the difference between a raw pretrained model and an instruction-tuned model is important for practitioners because they behave very differently in deployment. A raw pretrained model continues the statistical pattern of its training text, while an instruction-tuned model responds to the intent behind a user's message.
Catastrophic Forgetting and Mitigation
A risk of fine-tuning is catastrophic forgetting: as the model updates its weights to fit the new task, it may overwrite representations needed for general tasks learned during pretraining. For example, fine-tuning aggressively on a narrow medical QA dataset might degrade the model's ability to handle general language tasks or even reduce its factual accuracy outside the medical domain.
Several strategies mitigate this risk. Using a low learning rate during fine-tuning, much lower than the pretraining learning rate, limits how drastically weights change. Elastic weight consolidation applies a regularization term that penalizes changes to weights identified as important for the pretraining objective. Parameter-efficient fine-tuning methods like LoRA add new parameters rather than modifying existing ones, preserving the pretrained weights entirely and making forgetting structurally impossible for the frozen layers.
Evaluating fine-tuned models on both in-domain and general benchmarks is good practice to confirm that task-specific improvement has not come at the cost of general capability, especially when the fine-tuning dataset is small and domain-specific.