Back to Study Material Machine Learning

Logistic regression

Logistic regression is a supervised classification algorithm that predicts the probability that an example belongs to a particular class. Despite its name ...

Logistic regression

Classification Through a Probabilistic Lens

Logistic regression is a supervised classification algorithm that predicts the probability that an example belongs to a particular class. Despite its name it is a classifier, not a regression model. It models the log-odds of the positive class as a linear function of the features, then applies the logistic sigmoid function to convert that linear output into a probability between zero and one.

The sigmoid function has an S-shaped curve. Very large positive inputs produce probabilities close to one, very large negative inputs produce probabilities close to zero, and an input near zero produces a probability near 0.5. A threshold — commonly 0.5 — converts the probability into a binary label. Choosing a different threshold lets you trade off between precision and recall depending on the application.

Training uses maximum likelihood estimation rather than least squares. The algorithm finds weights that maximize the probability of observing the training labels given the features. In practice this is done by minimizing the log-loss, also called binary cross-entropy, using gradient descent or a closed-form solver for small datasets.

Logistic regression assumes that the decision boundary between classes is linear in feature space. This means it works well when classes are roughly linearly separable but fails when the boundary is curved or complex. Adding polynomial features can extend the model to non-linear boundaries at the cost of increased complexity and potential overfitting.

A concrete example: predicting whether an email is spam based on word frequencies. Logistic regression assigns a weight to each word. Words like "lottery" get a large positive weight pushing toward spam, while words like "meeting" get a negative weight. The sigmoid converts the weighted sum into a probability of spam.

Exam tip: know that logistic regression outputs a probability, not just a class, which makes it useful when the cost of false positives and false negatives differ — adjust the threshold, not the model.