Back to Study Material Machine Learning

Self-supervised learning

Self-supervised learning is a form of unsupervised representation learning where the model generates its own supervisory signal from the structure of the r...

Self-supervised learning

Creating Your Own Labels from Raw Data

Self-supervised learning is a form of unsupervised representation learning where the model generates its own supervisory signal from the structure of the raw data, without any human-provided labels. The model is trained on a pretext task whose labels can be derived automatically from the data itself. After pretraining, the learned representations are transferred to downstream tasks where labelled data may be scarce.

The pretext task is designed to require the model to understand meaningful structure in order to solve it. A language model that predicts the next word in a sentence must learn grammar, semantics, and world knowledge to succeed. A model trained to predict which of two image patches are from the same image must learn visual texture and object structure. Neither task requires a human annotator.

Contrastive learning is a prominent family of self-supervised methods. It trains the model to bring representations of similar examples close together in embedding space while pushing representations of dissimilar examples apart. SimCLR trains an image encoder by treating different augmented views of the same image as positives and different images as negatives. BYOL achieves similar results without negative pairs by using two networks with slightly different weights.

Language models like BERT use masked language modeling as a pretext task: some tokens in a sentence are hidden and the model must predict them from context. This forces the model to learn rich contextual representations. The resulting pretrained model transfers effectively to tasks like text classification, question answering, and named entity recognition with relatively little labelled fine-tuning data.

A concrete example: a self-supervised model pretrained on 100 million unlabelled medical records using a masked prediction pretext task was fine-tuned on only 5,000 labelled patient records and matched the performance of a fully supervised model trained on 100,000 labelled records.

Exam tip: distinguish self-supervised from semi-supervised learning. Self-supervised uses no human labels at all during pretraining; semi-supervised uses a small set of human labels alongside a larger unlabelled set throughout training.