|
Not a long time ago, most anti-spam products simply used a list of keywords to
identify spam. A good set of keywords could catch plenty of spam. However, the
keyword-based anti-spam filter requires manual updating and can be easily fooled
by tweaking the message a little. Spammers simply examine the latest anti-spam
techniques and find ways to bypass them. This results in a huge number of false
positives the keyword-based filter produces.
The vital issue to elaborate a new effective technique to fight against spam
stood
up. The experience showed that this new method must adapt itself to the
spammers'
tactics that change with time. So does the Bayesian filter.
The Bayesian filtering is based on the principle that most events are dependent
and
that the probability of an event occurring in the future can be inferred from
the
occurrences of this event in the past. This same approach is used to identify
spam.
If some piece of text occurred mostly in spam emails but not in legitimate mail,
then
it would be reasonable to suppose that this email is probably spam.
To effectively filter mail using the Bayesian technology, the user needs to
generate a database of words collected from spam and legitimate mail. Then a
probability value is assigned to each word; the probability is based on the
calculations that take into account how often that word occurs in spam as
opposed to legitimate mail.
After the legitimate and spam databases are created during an initial training
period, the word probabilities can be calculated and the Bayesian filter is
ready for use. When a new mail arrives, it is broken into words and the most significant
words are singled out. From these words, the Bayesian filter calculates the
probability of a new message being spam or not. If the probability is greater than a spam
threshold, say 0.9, then the message is classified as spam.
It is important to note that the analysis of spam and legitimate mail is
performed on the mail the particular user (organization, company, etc.) receives, and
therefore the Bayesian filter is adjusted to this particular person, company, or
organization. For example, a financial institution might receive a lot of emails with the word
"mortgage" and would get a lot of false positives if using an outdated antispam filter. The Bayesian filter analyzes the entire message with the word
"mortgage", and concludes whether this email is spam or legitimate basing NOT
only on a single keyword "mortgage". The Bayesian approach to filter spam is highly effective -
spam detection rates of over 99.7% can be achieved with a very low number of
false positives!
Advantages of the Bayesian filter:
- The Bayesian filtering is a much more intelligent approach because it
examines all aspects of a message, as opposed to keyword checking that
classifies a mail as spam on the basis of a single word.
- The Bayesian filter is constantly self-adapting - by learning from new spam
and
new valid inbound mails, the Bayesian filter evolves and adapts to new spam
techniques.
- The Bayesian technique is sensitive to the user – it learns the email habits
of the
company and understands that, for example, the emails with the word "mortgage"
are not always spam.
- The Bayesian method is multi-lingual and international - being adaptive it
can be
used for any language. The Bayesian filter also takes into account certain
languages
deviations or the diverse usage of certain words in different areas, even if the
same
language is spoken.
- The Bayesian filter is difficult to fool, as opposed to a keyword filter - an
advanced spammer who wants to trick the Bayesian filter can either use fewer
words that usually indicate spam, or more words that generally indicate
valid mail (such as a valid contact name, etc). Doing the latter is impossible
because the spammer would have to know the email profile of each recipient
- and a spammer can never hope to gather this kind of information from every
intended recipient.
|