Naive Bayes
Naive Bayes is a probabilistic classifier built on Bayes' theorem with a strong but often useful simplifying assumption: that all features are conditionall...
Probabilistic Classification with a Useful Simplification
Naive Bayes is a probabilistic classifier built on Bayes' theorem with a strong but often useful simplifying assumption: that all features are conditionally independent given the class label. Despite this assumption being almost never literally true, Naive Bayes performs surprisingly well in practice for certain types of problems.
Bayes' theorem gives the probability of a class given a set of observed features as proportional to the probability of those features given the class multiplied by the prior probability of the class. To compute the probability of all features together given the class, Naive Bayes multiplies the individual feature probabilities — this is only valid if the features are independent, hence the "naive" label.
The algorithm is fast to train (a single pass through the data to compute probabilities) and requires very little data to produce reasonable estimates. It scales naturally to high-dimensional feature spaces because each feature's probability is estimated independently. These properties make it well-suited to text classification, where the number of unique words (features) can be enormous.
Three main variants handle different feature types. Gaussian Naive Bayes assumes numerical features follow a Gaussian distribution. Bernoulli Naive Bayes works with binary features — word presence or absence. Multinomial Naive Bayes uses word counts and is the standard choice for text document classification.
A concrete example: spam filtering with Multinomial Naive Bayes. The model estimates the probability of each word appearing in spam and in legitimate email. A new email's words are evaluated against these probabilities. A message containing "winner," "claim," and "prize" gets a high spam probability despite the independence assumption being violated — these words are correlated in spam, but the classifier still works well.
Exam tip: be ready to explain the independence assumption and why the algorithm works despite it being false. The short answer is that the decision boundary can still be correct even when probability estimates are inaccurate.