Technology
A Plan for Spam
paulgraham.com
Technology
20 min read → 80 sec
Bayesian content filtering can identify spam from the words and headers in each message, with far fewer false positives than hand-built rules. Per-user training and probability-based scoring make the filter adapt as spam changes.
Bayesian filtering stops spam better than hand-written rules
Statistical learning adapts as spam evolves and finds signals rules miss
2 / 19
A tweaked Bayesian filter misses fewer than 5 per 1000 spams, with 0 false positives
False positives are far worse than missed spam, so filters bias strongly against them
3 / 19
Per-user Bayesian filters work better because each user has distinct good and bad word patterns
4 / 19
Simple rule baseline
79.7%
Spam caught by a rule for the word "click", with 1.2% false positives
5 / 19
Hand-built feature filters get much harder on the last few percent and raise false positives
6 / 19
Statistical filtering finds unexpected spam signals like "per", "FL", and "ff0000"
Bayesian filters discover patterns rule-writers would never think to encode
7 / 19
The filter uses about 4000 spam messages and 4000 nonspam messages as training corpora
8 / 19
Token counts scan full messages, including headers, html, and javascript, not just visible text
9 / 19
The algorithm applies three adjustments to raw token counts
1
Doubles nonspam counts to bias against false positives
2
Ignores rare words that appear in too few messages
3
Assigns unseen-word probabilities of .01 and .99
10 / 19
New mail is scored from the 15 most interesting tokens, and spam is flagged above .9 probability
11 / 19
Probabilities are clearer than spam scores, because evidence combines by Bayes' Rule into explicit odds
12 / 19
Words rare in spam, like "though" or "apparently", lower spam probability as much as bad words raise it
13 / 19
Each user should train the filter with delete and delete-as-spam actions to build personal probabilities
14 / 19
Whitelists help mainly to save computation, because unknown addresses can still belong to trusted senders
15 / 19
Bayesian filters adapt as spam evolves, because spammers must still deliver recognizable messages
Spammers could evade checksum filters with mad-lib variation, but Bayesian filters force messages toward ordinary mail
16 / 19
Spam economics
15 per million
Response rate spam works on, because sending 1 million emails costs about $200
17 / 19
Filtering out 95% of spam raises the cost to reach a given audience by a factor of 20
18 / 19
19 / 19
Carouse
Turn any article, video, or podcast into a carousel you can browse in 30 seconds.