Back to Study Material Machine Learning

Outliers

Outliers are data points that differ substantially from the rest of the dataset. They can arise from genuine extreme events, measurement errors, data entry...

Outliers

Extreme Values and What to Do with Them

Outliers are data points that differ substantially from the rest of the dataset. They can arise from genuine extreme events, measurement errors, data entry mistakes, or system anomalies. Deciding whether to remove, transform, or keep an outlier is a nuanced decision that depends on its cause and the modeling goal.

Outliers matter because many machine learning algorithms are sensitive to them. Linear regression minimizes squared error, which means a single extreme point can pull the fitted line dramatically toward itself. Algorithms that compute distances — like K-nearest neighbors or K-means — treat outliers as being very far from other points, which distorts cluster structures and neighbor relationships. Neural networks trained with gradient descent can also be destabilized by extreme values.

Detection methods range from simple to sophisticated. The interquartile range method flags values more than 1.5 times the IQR below the first quartile or above the third quartile. Z-scores flag values more than three standard deviations from the mean. Isolation Forest, a machine learning method, detects outliers by measuring how easily a point can be isolated from the rest of the dataset.

Once detected, options include removal (appropriate for genuine errors), capping (winsorizing extreme values to a maximum allowed threshold), log transformation (which compresses the scale of right-skewed distributions and reduces the influence of large values), and keeping them if they represent real and important edge cases.

A concrete example: in a dataset of employee salaries, the CEO earns 200 times the median salary. This is a genuine value, not an error, but it will distort a mean-imputed salary column and inflate variance. A log transformation or median-based approach handles it more robustly.

Exam tip: the key question is always whether the outlier reflects a real phenomenon or a data quality problem. The answer determines the treatment.