K-nearest neighbors
K-nearest neighbors is one of the simplest supervised learning algorithms. To predict the class or value of a new point, it finds the K training examples c...
Classify by Looking at Your Neighbors
K-nearest neighbors is one of the simplest supervised learning algorithms. To predict the class or value of a new point, it finds the K training examples closest to that point in feature space and returns either the majority class among those neighbors for classification or their average value for regression. There is no explicit training phase: the algorithm simply stores all training examples and defers computation to prediction time, which is why it is called a lazy learner.
The choice of K is the primary hyperparameter. A small K, such as one, makes the model sensitive to individual training examples and noise, leading to jagged decision boundaries that may overfit. A large K smooths the boundary by averaging over more neighbors, reducing variance at the cost of increased bias. The optimal K is typically found through cross-validation on the training set.
A critical requirement is that features be on comparable scales. If one feature measures age in years and another measures income in tens of thousands, the distance between two points will be dominated almost entirely by the income difference. Standardizing or normalizing features before applying KNN ensures all features contribute equally to the distance calculation.
Prediction is slow when the training set is large because every query requires computing distances to all training points. Data structures such as KD-trees and ball-trees organize the training data to accelerate nearest-neighbor lookup, reducing query time from linear to logarithmic in most practical cases.
A concrete example: in a fruit classification task, a new fruit with medium roundness and orange color is queried. Its three nearest neighbors are all oranges, so the model predicts orange regardless of the precise feature values, purely based on similarity.
Exam tip: know that KNN is non-parametric, meaning it makes no assumption about the underlying data distribution, but becomes computationally expensive at prediction time as the training set grows.