Back to Study Material Machine Learning

Distance metrics

Distance metrics are mathematical functions that quantify how far apart two data points are in feature space. Algorithms like KNN, K-means clustering, and ...

Distance metrics

Measuring How Different Two Points Are

Distance metrics are mathematical functions that quantify how far apart two data points are in feature space. Algorithms like KNN, K-means clustering, and DBSCAN rely entirely on distances to define similarity, so the choice of metric directly shapes which patterns the algorithm finds. A poor metric can make genuinely similar points appear distant and vice versa.

Euclidean distance is the straight-line distance between two points, calculated as the square root of the sum of squared differences across all dimensions. It is intuitive and widely used, but it treats all dimensions equally and is sensitive to scale. If features are not normalized, dimensions with larger numerical ranges will dominate the distance.

Manhattan distance, also called the L1 norm, sums the absolute differences along each dimension rather than squaring them. It is less sensitive to outliers than Euclidean distance because squaring amplifies large individual differences. Manhattan distance is often preferred in high-dimensional spaces because the curse of dimensionality causes Euclidean distances to become increasingly similar across all pairs of points.

Cosine similarity measures the angle between two vectors rather than the geometric distance between their endpoints. It captures the orientation of a vector, making it ideal for text data where a document's length should not affect its similarity to other documents: a short article and a long article on the same topic should be considered close even though their word-count vectors differ in magnitude.

Minkowski distance is a generalization that includes both Euclidean and Manhattan distance as special cases, with a parameter p controlling the behavior.

A concrete example: for a spam detection system using word frequency features, cosine similarity correctly identifies two emails as similar even if one is much longer than the other, because similarity is determined by which words appear proportionally rather than by absolute counts.

Exam tip: in interviews, explain when to prefer Manhattan over Euclidean — Manhattan is more robust in high dimensions and when features have outliers, while Euclidean is natural when direction and magnitude both matter.