Better Bayesian Filtering

paulgraham.com

Technology

18 min read → 65 sec

Bayesian spam filtering works far better when it uses email-specific signals, richer tokenization, and per-user training. Results from late 2002 to early 2003 show high detection, low false positives, and limits of network-level filtering.

Bayesian spam filtering reaches 99.5% detection with under .03% false positives

Smarter tokenization and email-specific signals close the gap to CRM114

Pantel and Lin achieved only 92% detection and 1.16% false positives from weaker design choices

They trained on just 160 spam and 466 nonspam mails, too little data

Spam filters should use email-specific signals, because headers, structure, and adversaries matter

Ignoring message headers throws away strong spam signals and hurts performance

Preserving case and marking context creates stronger tokens like Subject*FREE with probability 0.9999

Stemming tokens is premature optimization that weakens discrimination when corpora grow

Using all tokens lets long spam and random filler dilute the spam signal

Degeneration backs off missing tokens to less specific forms and picks the probability farthest from .5

Thresholds now use .9999 and .0001, with extra scaling for tokens seen only in one corpus

Biasing against false positives by double-counting nonspam tokens gives a useful tradeoff knob

HTML should be partially parsed, because links and images help, but full parsing overfits HTML

False positives from statistical filters tend to resemble spam, making them less costly than blacklist errors

The hardest remaining spam mimics normal personal mail or overlaps with the recipient's real vocabulary

Per-user statistical filters are more durable than network-level filters against spammer adaptation

Individual filters slow spammers' edit-compile-test loop because every user's probabilities differ