Back to Study Material Machine Learning

One-hot encoding

One-hot encoding converts a categorical feature with N distinct values into N binary columns. Each column corresponds to one category and contains a 1 if t...

One-hot encoding

Turning Categories into Binary Flags

One-hot encoding converts a categorical feature with N distinct values into N binary columns. Each column corresponds to one category and contains a 1 if the example belongs to that category and a 0 otherwise. Only one column is active at a time per example, hence the name one-hot. This representation removes any artificial ordinal relationship among the categories, making it the default choice for nominal features used with linear models or neural networks.

The major downside is dimensionality. A feature representing the nationality of users might have 200 distinct values, producing 200 binary columns. When many such features exist or when categories are very numerous, the feature matrix becomes very wide and sparse, increasing memory usage and computation time. Regularization, dimensionality reduction, or alternative encodings such as target encoding may be needed for very high-cardinality features.

A related issue is the dummy variable trap. In linear regression, if you include all N one-hot columns, any one of them is perfectly predictable from the rest, creating perfect multicollinearity. The standard fix is to drop one category, using N minus one columns instead. The dropped category becomes the reference category implicitly captured by the intercept.

Most modern machine learning libraries handle the dummy variable trap automatically when the appropriate parameter is set, but it is important to know why it exists.

A concrete example: a feature called animal_type with values cat, dog, and bird becomes three columns: is_cat, is_dog, and is_bird. An example representing a dog gets the vector 0, 1, 0. A cat gets 1, 0, 0. The model treats each animal type independently with no implied ordering between them.

Exam tip: in interviews, explain that one-hot encoding is appropriate for nominal categories with linear models, but that very high-cardinality features can cause dimensionality problems and may require alternatives such as embedding layers in neural networks.