Back to Study Material Machine Learning

Principal Component Analysis

Principal Component Analysis, or PCA, is a dimensionality reduction technique that transforms a dataset into a new coordinate system where the axes — cal...

Principal Component Analysis

Compressing Information While Preserving Meaning

Principal Component Analysis, or PCA, is a dimensionality reduction technique that transforms a dataset into a new coordinate system where the axes — called principal components — are ordered by the amount of variance they capture. The first principal component captures the greatest variance, the second captures the most remaining variance while being orthogonal to the first, and so on.

High-dimensional data presents practical challenges: it is computationally expensive to process, difficult to visualize, and prone to the curse of dimensionality — the tendency for distances to become meaningless and models to overfit as dimensions increase. PCA addresses this by projecting data onto a lower-dimensional subspace that retains as much variability as possible.

Mathematically, PCA computes the eigenvectors and eigenvalues of the data's covariance matrix. The eigenvectors define the directions of the principal components, and the eigenvalues indicate how much variance each component captures. A scree plot of eigenvalues helps identify how many components to retain — often a small number of components captures 90 to 95 percent of total variance.

PCA requires data to be centered (mean-subtracted) and typically standardized before application. Otherwise features with larger scales dominate the principal components regardless of their actual information content.

PCA is useful for visualization: projecting data onto its first two principal components allows plotting of high-dimensional data in two dimensions. It is also used as a preprocessing step to reduce noise, speed up downstream algorithms, and mitigate the curse of dimensionality.

A concrete example: a dataset of 1,000 gene expression measurements per patient is reduced to 20 principal components that capture 92 percent of variance. A logistic regression trained on 20 components trains in seconds and avoids overfitting that plagued the 1,000-dimensional version.

Exam tip: PCA finds directions of maximum variance, not maximum discrimination between classes. For supervised dimensionality reduction that maximizes class separability, use Linear Discriminant Analysis instead.