Pretraining
Pretraining refers to training a model on a large, general dataset with a self-supervised objective before adapting it to a specific downstream task. The p...
What Pretraining Means in Deep Learning
Pretraining refers to training a model on a large, general dataset with a self-supervised objective before adapting it to a specific downstream task. The pretraining phase teaches the model broadly useful representations: for language models, this means learning grammar, world knowledge, and reasoning patterns from vast text corpora; for vision models, it means learning to recognize shapes, textures, and objects from large image datasets.
The core advantage of pretraining is that the model does not need task-specific labeled data to acquire foundational capabilities. Text corpora scraped from the web contain billions of tokens, and the next-token prediction objective provides a dense training signal at every token position without human labeling. Once pretrained, a model can be adapted to specific tasks with far less data than would be required to train from scratch.
Exam tip: Distinguish pretraining from transfer learning in general. Transfer learning is the broader concept of reusing representations learned on one task for another task. Pretraining specifically refers to the initial large-scale unsupervised or self-supervised training phase, which is then followed by fine-tuning. All pretrained models benefit from transfer learning, but not all transfer learning involves a pretraining phase in this sense.
Pretraining Objectives for Language and Vision
Language model pretraining uses two main objectives. Causal language modeling, used by GPT-style models, trains the model to predict the next token given all preceding tokens and processes the sequence left to right. Masked language modeling, used by BERT, randomly masks tokens in the input and trains the model to predict them, allowing the model to use both left and right context simultaneously but preventing direct use for autoregressive generation.
Vision model pretraining has evolved significantly. Early approaches used supervised pretraining on ImageNet with human-provided class labels, which despite being supervised is still called pretraining when the representations are intended for transfer. Contrastive learning methods like SimCLR and CLIP learn representations by bringing together semantically similar examples and pushing apart dissimilar ones, without requiring explicit class labels. Masked image modeling methods inspired by BERT mask patches of images and train the model to reconstruct them.
The quality and diversity of the pretraining data often matters more than the specific objective for downstream task performance, a finding that has driven investment in large-scale data curation and filtering pipelines.
Compute Requirements and Environmental Considerations
Pretraining large foundation models requires massive compute resources. Training GPT-3 was estimated to consume thousands of petaflop-days of computation, equivalent to running billions of floating-point operations per second continuously for thousands of days. This requires specialized hardware clusters with hundreds or thousands of GPUs or TPUs, interconnected with high-bandwidth networking to coordinate gradient updates across devices.
The financial cost of a single large-scale pretraining run ranges from hundreds of thousands to tens of millions of dollars, which concentrates the capacity to train frontier models in a small number of well-resourced organizations. This concentration has prompted discussion about accessibility, reproducibility, and the environmental impact of training large models.
For practitioners who do not have access to frontier-scale compute, the practical implication is to rely on publicly released pretrained checkpoints and invest resources in fine-tuning rather than pretraining from scratch. Understanding when pretraining from scratch is justified, such as for highly domain-specific data distributions not covered by existing models, is an important judgment call in applied deep learning work.