Naive Bayes, in depth: from Bayes' rule to a calibrated text classifier, with a worked spam example

From Bayes' rule to the naive assumption

We want the class k that is most probable given the features x = (x1, ..., xn). Bayes' rule gives P(k | x) = P(k) P(x | k) / P(x). The denominator is the same for every class, so for a decision we only need the numerator. The hard part is P(x | k), a joint distribution over possibly tens of thousands of word features, which no dataset can estimate directly. The naive assumption factorises it:

P(x | k) = P(x1 | k) * P(x2 | k) * ... * P(xn | k)

predict(x) = argmax_k [ log P(k) + sum_i log P(xi | k) ]

Each factor is now a one-dimensional distribution that can be estimated from counts. The number of parameters falls from exponential in n to linear: K classes times V vocabulary terms. That is why naive Bayes trains on millions of documents in seconds, and why it works with very little data. A few hundred labelled examples give usable estimates for the common words.

In log space the model is linear. The score for class k is a bias, log P(k), plus a weight, log P(w | k), for every token occurrence. For two classes, the decision depends only on the difference of those weights, the per-word log-likelihood ratio. Naive Bayes and logistic regression are the same family of linear classifier fitted in two different ways. NB fits each weight separately from counts. Logistic regression fits all the weights jointly by optimising the conditional likelihood.

Choosing an event model

'Naive Bayes' is a family. The member is chosen by how you model P(xi | k):

The Bernoulli model differs in one important way. It multiplies in a factor of 1 - P(w | k) for every vocabulary word that is absent, so long documents are penalised by thousands of absences. That suits short messages and works badly on long articles. The multinomial model ignores absent words and counts repeats. The Gaussian model assumes each feature is normal within each class. It breaks on heavy-tailed or multi-modal features, and binning or a log transform often helps it more than any other tuning.

Training is counting

Training is a single pass of counting. For each class k, count documents (for the prior) and count each word's occurrences, N(k, w), and the class's total tokens, N(k). The smoothed likelihood is (N(k, w) + alpha) / (N(k) + alpha V). With alpha = 1 this is Laplace smoothing, which pretends every word was seen once more in every class. A from-scratch version:

import math
from collections import Counter, defaultdict

class MultinomialNB:
    def __init__(self, alpha=1.0):
        self.alpha = alpha

    def fit(self, docs, labels):
        self.docs = Counter(labels)
        self.counts = defaultdict(Counter)          # class -> word -> count
        for tokens, k in zip(docs, labels):
            self.counts[k].update(tokens)
        self.vocab = set().union(*self.counts.values())
        V = len(self.vocab)
        n = sum(self.docs.values())
        self.log_prior = {k: math.log(c / n) for k, c in self.docs.items()}
        self.log_lik = {}
        for k, cnt in self.counts.items():
            denom = sum(cnt.values()) + self.alpha * V
            self.log_lik[k] = {w: math.log((cnt[w] + self.alpha) / denom) for w in self.vocab}
        return self

    def predict_log_scores(self, tokens):
        known = [w for w in tokens if w in self.vocab]   # drop out-of-vocabulary words
        return {k: self.log_prior[k] + sum(self.log_lik[k][w] for w in known)
                for k in self.log_prior}

    def predict_proba(self, tokens):
        s = self.predict_log_scores(tokens)
        m = max(s.values())                              # log-sum-exp for stability
        z = sum(math.exp(v - m) for v in s.values())
        return {k: math.exp(v - m) / z for k, v in s.items()}

Two implementation rules are not optional. Work in log space. A product of a few hundred probabilities near 0.001 underflows double precision to zero for every class, and every prediction becomes a tie. Normalise with log-sum-exp. Subtract the maximum score before exponentiating, as above. This is the same trick as a stable softmax. Because the counts are additive, training on shards and summing the count tables gives exactly the model you would get from one pass. That makes NB trivial to distribute and to update online.

Worked example: a tiny spam filter

Train on eight messages: four spam ('win cash now', 'win a free prize now', 'free cash offer', 'claim your free prize') and four ham ('meeting moved to noon', 'lunch at noon', 'project meeting notes', 'free for lunch now'). The vocabulary has V = 18 words. The spam class has 15 tokens and ham has 14. Priors are both 0.5. Classify 'free cash meeting now' with Laplace smoothing:

Each likelihood is (count + 1) / (class tokens + 18). 'meeting' never appeared in spam, but smoothing still gives it 0.0303 instead of zero. Without smoothing, that single word would veto the spam class no matter what else the message contained. Summing logs gives a spam score of -11.096 and a ham score of -12.071. The log-ratio column adds up to the difference, +0.976, and the priors cancel. The message is classified as spam with posterior 0.726. 'free', 'cash' and 'now' each point to spam, and 'meeting' pulls the other way but cannot outvote them.

Read the table as an explanation. Each word's log ratio is its contribution to the decision. This per-feature transparency is a real operational advantage. When a support ticket asks why a message was flagged, you can list the words that did it.

Why it works when the assumption is wrong

Words in real text are strongly dependent. 'New' and 'York' occur together, and so do 'free' and 'prize'. Why does a model that assumes otherwise classify well? Domingos and Pazzani (1997) showed that under zero-one loss, naive Bayes can be optimal even when independence is badly violated. Classification needs only the right class to have the highest score, not correct probabilities. Dependencies inflate or deflate the scores of all classes, often in the same direction, and the ranking survives.

The cost shows up in the probabilities. Correlated words are counted as independent evidence, so a phrase repeated three times or three synonyms in a row multiply the evidence three times. NB posteriors pile up near 0 and 1. A posterior of 0.999 might really mean 0.85. If downstream logic uses the probability (thresholds, expected-cost routing, ranking by confidence), calibrate it first.

Calibrating the probabilities

Fit a calibrator on held-out data. Platt scaling (a sigmoid on the score) works when the distortion is monotone and smooth. Isotonic regression is more flexible but needs more data. In scikit-learn the whole pipeline is a few lines:

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB, ComplementNB
from sklearn.calibration import CalibratedClassifierCV
from sklearn.pipeline import make_pipeline

nb = make_pipeline(CountVectorizer(ngram_range=(1, 2), min_df=2),
                   CalibratedClassifierCV(ComplementNB(alpha=0.5), method="isotonic", cv=5))
nb.fit(train_texts, train_labels)
proba = nb.predict_proba(test_texts)        # calibrated probabilities

Check the result with a reliability diagram and expected calibration error, as described in model calibration. Tune alpha on validation data instead of taking the default. Values between 0.01 and 1 are common, and large vocabularies with rare words often want smaller alpha.

Complement NB, TF-IDF and feature hashing

Rennie et al. (2003), Tackling the Poor Assumptions of Naive Bayes Text Classifiers, proposed fixes that are now standard. Complement NB estimates each class's weights from all the other classes' data, which steadies the estimates for small classes and helps on imbalanced data. TF-IDF and length normalisation damp the effect of repeated words and long documents. scikit-learn's MultinomialNB accepts fractional counts, so TF-IDF features work directly.

Feature hashing bounds memory for streaming or huge vocabularies. It maps each token to one of 2^20 columns, with no vocabulary to store, and the model updates with partial_fit(X, y, classes=...). There is a trap. HashingVectorizer defaults to alternate_sign=True, which gives half the features negative values so that collisions cancel. MultinomialNB rejects negative input. Set alternate_sign=False for NB. If you need count-style sketches in other parts of the system, Count-Min Sketch uses the same hash-into-buckets idea.

Failure modes

What goes wrong in production:

• Zero probabilities from alpha = 0, or from a word seen only in one class. One token vetoes a class. Always smooth.
• Underflow from multiplying raw probabilities. Use logs everywhere.
• Tokeniser drift. Train and serve must share one tokeniser and normalisation (case folding, Unicode, URL handling). Otherwise serving words miss the vocabulary and are silently dropped.
• Prior shift. Priors learned from a 50/50 training set misstate a 2% production base rate. MultinomialNB and BernoulliNB accept class_prior. ComplementNB ignores it, so move the decision threshold instead.
• Adversarial padding. Spammers append innocent text to drown the signal, and Bernoulli models are especially exposed. Combine NB with features it cannot be padded against (sender reputation, URLs).
• Overconfident scores used as probabilities. Calibrate them first.
• Gaussian NB on skewed features, or forgetting that var_smoothing floors near-zero variances.

When to use it

Ng and Jordan (2001) compared naive Bayes with logistic regression and found the classic pattern. NB approaches its (higher) asymptotic error with far fewer examples, and logistic regression wins once data is plentiful. So reach for NB when labels are scarce, training must take seconds, the model must update online, or every decision must be explainable word by word. Typical uses are spam and abuse triage, routing tickets to queues, language and topic detection, and baselines that every fancier model must beat. With a few thousand labels, a regularised logistic regression on the same features usually overtakes it. With more, a small fine-tuned model captures word order and context that no bag-of-words model sees. On tabular data, compare it with decision trees, which model interactions that NB assumes away. The email spam architecture article shows where a fast NB-style filter fits in a larger pipeline.

What to do next

1. Build a CountVectorizer plus MultinomialNB baseline on your labelled text and record macro-F1.
2. Try ComplementNB and Bernoulli NB on the same split, and keep whichever validates best.
3. Grid-search alpha from 0.01 to 1 on a log scale.
4. Plot a reliability diagram. If you use probabilities, wrap the model in CalibratedClassifierCV.
5. Set class priors or the decision threshold to the production base rate, not the training mix.
6. Freeze the tokeniser as a versioned artifact shared by training and serving.
7. If the vocabulary is unbounded, switch to HashingVectorizer with alternate_sign=False and partial_fit.
8. Train a logistic regression on the same features and keep NB only where it wins or where its speed or explainability matters.