CRM114 (program)

From Wikipedia, the free encyclopedia

CRM114 (full name: "The CRM114 Discriminator") is a program based upon a statistical approach for classifying data, and especially used for filtering email spam. While others have done statistical Bayesian filtering based upon the frequency of single word occurrences in email, CRM114 achieves a higher rate of spam recognition through creating hits based upon phrases up to five words in length. These phrases are used to form a Markov Random Field representing the incoming texts. With this additional contextual recognition, it is one of the more accurate spam filters available. The author claims recognition rates as high as 99.87% in some situations; Holden ^[1] and TREC 2005 and 2006.^[2] ^[3] gave results of better than 99%, with significant variation depending on the particular corpus. CRM114's classifier can also be switched to use Littlestone's Winnow algorithm, character-by-character correlation, a variant on KNN (K-nearest neighbor algorithm) classification called Hyperspace, a bit-entropic classifier that uses entropy encoding to determine similarity, and other more experimental classifiers.

As an example of pattern recognition software, CRM114 is a good example of machine learning accomplished with a reasonably simple algorithm. GPLed Source code in C is available through the external link.

At a deeper level, CRM114 is also a string pattern matching language, similar to grep or even Perl; although it is Turing complete it is highly tuned for matching text, and even a simple (recursive) definition of the factorial takes almost ten lines, looking somewhat confusing to the uninitiated. Part of this is because the crm114 language syntax is not positional, but declensional. As a programming language, it may be used for many other applications aside from detecting spam. CRM114 uses the TRE approximate-match regex engine, so it's possible to write programs that do not depend on absolutely identical strings matching to function correctly.

[edit] Origin of the name

The name CRM114 is taken from the Stanley Kubrick movie Dr. Strangelove or: How I Learned to Stop Worrying and Love the Bomb (1964). The piece of radio equipment onboard the B-52 which is designed not to receive messages lacking a specific code-prefix is designated as the "CRM-114 Discriminator." Kubrick has used this designation in other films as well.