Semi-supervised learning
Semi-supervised learning is a machine learning paradigm that combines a small amount of labelled data with a large amount of unlabelled data during trainin...
Learning from a Mix of Labelled and Unlabelled Data
Semi-supervised learning is a machine learning paradigm that combines a small amount of labelled data with a large amount of unlabelled data during training. Labelling data is expensive and time-consuming: it requires domain experts to review examples and assign correct labels. Unlabelled data, by contrast, is often abundant and cheap to collect. Semi-supervised learning extracts useful structure from unlabelled data to improve models trained on limited labels.
The underlying assumption is that the structure of the unlabelled data — the clusters, manifolds, and density patterns in feature space — correlates with the label boundaries. If two points are close in feature space and the label boundary should not cross between them, the model should classify them the same way even if only one is labelled. This is the smoothness assumption.
Self-training is the simplest semi-supervised algorithm. A model is trained on the small labelled set, then used to predict labels for the unlabelled examples with high confidence. Those pseudo-labelled examples are added to the training set, and the model is retrained. The process repeats, gradually incorporating more examples. The risk is error propagation: early wrong pseudo-labels can mislead later iterations.
Label propagation treats data as a graph where edges connect similar examples. Labels from labelled nodes spread to unlabelled neighbors according to edge weights, producing smooth label assignments consistent with the local structure of the data.
Consistency regularization, used in methods like MixMatch and FixMatch for image classification, trains the model to produce the same prediction for an unlabelled example and its augmented versions, enforcing stable predictions without requiring explicit labels.
A concrete example: a medical imaging model for detecting a rare condition has only 200 labelled scans but access to 10,000 unlabelled scans. Semi-supervised training using consistency regularization improves classification accuracy from 74 to 83 percent by leveraging the structural information in the unlabelled images.
Exam tip: be ready to explain the cluster assumption, smoothness assumption, and manifold assumption that motivate semi-supervised learning, and give one concrete algorithm for each approach.