Support vector machine
A support vector machine is a supervised learning algorithm that finds the hyperplane separating two classes with the maximum possible margin. The margin i...
Finding the Widest Possible Boundary
A support vector machine is a supervised learning algorithm that finds the hyperplane separating two classes with the maximum possible margin. The margin is the distance from the hyperplane to the nearest training examples of each class. Those nearest examples are called support vectors because they support, or define, the boundary. Maximizing the margin makes the model more robust: small perturbations in new data points are less likely to push them across the decision boundary.
When the classes are not perfectly separable, a soft-margin SVM introduces slack variables that allow some examples to fall inside the margin or even on the wrong side. A regularization parameter C controls the trade-off between maximizing the margin and penalizing these violations. A large C tolerates few violations and fits the training data more tightly, risking overfitting. A small C allows more violations and produces a wider, more general margin.
SVM training solves a convex quadratic optimization problem, which guarantees that the optimal solution is globally optimal rather than a local minimum. This is a significant advantage over neural networks, where training can converge to local minima.
The algorithm naturally extends to multi-class problems using one-versus-one or one-versus-rest strategies, training a separate binary classifier for each pair or each class against all others.
A concrete example: classifying handwritten digits as threes or eights. An SVM finds the boundary in feature space that keeps both digit types as far from the boundary as possible. Only a handful of the most ambiguous digit images — the support vectors — determine where the boundary sits; all other training examples could be removed without changing it.
Exam tip: know that SVMs are effective in high-dimensional spaces, such as text classification where each word is a feature, and are memory-efficient during prediction because only support vectors need to be stored.