Challenge-response spam filtering
From Wikipedia, the free encyclopedia
Challenge/response or C/R systems are a type of spam filter that responds to incoming e-mail messages by sending a challenge to the claimed sender of the e-mail. The sender may then perform some action outlined in the challenge message to assure delivery of their message, which otherwise will not be delivered or may be delivered with a lower priority. These strategies are currently controversial among e-mail programmers and system administrators since spammers often forge the e-mail addresses of the claimed sender and so these challenges often go to the wrong people, creating e-mail backscatter. Even when the challenge is sent to the real sender, they often dislike the challenges on mail they feel should have been easily classified as non-spam and delivered.
Some Challenge/Response systems use other spam filtering techniques, such as whitelists, blacklists, or content filtering, to try to determine if the e-mail is spam or not, and only send a challenge if it can not make a clear determination. Almost all Challenge/Response systems will whitelist e-mail addresses that the user has sent e-mail to, so that these people will not receive a challenge when they reply.
[edit] Challenge/Response Verification Mechanisms
C/R systems attempt to provide challenges that can not easily be responded to by an automated system. In many cases they rely on a simple challenge that will be changed if spammers ever build such automated responders. Other systems attempt to produce challenges for which autoresponse is very difficult, or even an unsolved Artificial Intelligence problem.
One example of a challenge in a C/R system (also found in many web sites) is a "CAPTCHA" test, in which a mail sender is required to view an image containing a word or phrase, and respond with that word or phrase in text. The purpose of this is to ensure that automated systems (incapable of reading the image) cannot transmit email.
Another example is simply reading natural language instructions on how to reply, with the inclusion of a special string or pass-code in the reply. Other Turing Test approaches include a simple problem, or answering a simple question about the text or the recipient.
Some systems include a web URL, which can be loaded in an appropriate web browsing tool to respond to the challenge. In early systems, simply clicking on the link was sufficient to respond to the challenge, or simply sending any reply to the message.
A variant (not strictly challenge/response) involves sending a message indicating the original message has not been delivered, with instructions on a different means to get the message delivered.
[edit] Best Practices
C/R systems should, at a very minimum, comply with the requirements and recommendations of RFC 3834: Recommendations for Automatic Responses to Electronic Mail.
Brad Templeton maintains a detailed list of principles that C/R systems should obey: http://www.templetons.com/brad/spam/challengeresponse.html
Some C/R systems allow for the creation of "tagged" addresses or allow pass-codes placed in either the Subject: line or the body of the email, any of which allow messages to be accepted without being challenged. For example the Tagged Message Delivery Agent (TMDA) system can create "tagged" addresses that permit mail sent from a particular address, mail that contains a certain "keyword" or mail that is sent within a pre-set length of time, such as a day, a month, or a year. For example, a time-limited tagged address can be created for a short amount of time to allow correspondence related to an online order, such as shipping notices, but expire after a while to disallow future marketing e-mail from the online store without the store responding to message challenges.
Problems with sending challenges to forged e-mail addresses can be greatly reduced by not creating a new message that contains the challenge. Instead, the challenge can be placed in the Bounce message when the receiving mail system gives a rejection-code during the SMTP session. When the receiving mail system rejects an e-mail this way, it is the sending system that actually creates the bounce message. As a result, the bounce message will almost always be sent to the real sender, and it will be in a format and language that the sender will usually recognize.
Problems with sending challenges to forged e-mail addresses can also be reduced if the challenges are only sent when the from e-mail address has passed E-mail authentication. Systems such as SPF validate the correct from address directly, other systems such as DomainKeys can only be used when the envelope from matches the validated From: e-mail header.
[edit] Criticisms
Critics of C/R systems have raised several issues regarding their usefulness as an email defense. A number of the issues relate to all programs which autorespond to E-mail, including mailing list managers, vacation programs and bounce messages from mail servers.
Most criticisms have arisen due to implementations which send challenges inappropriately. In particular, a common early strategy was to challenge all mail from unknown senders, with known senders, including past valid correspondents, stored on a "whitelist." In addition, many programs send challenges on messages which are obviously spam or viruses (and thus often have a forged return address.)
[edit] Challenges sent to forged e-mail addresses
- Spammers and viruses send forged messages -- email with other people's addresses as sender address fields in the header (e.g. the From: field). Forged mail from valid addresses is becoming increasingly common as callback verification is increasingly used to detect spam. Most C/R systems challenging a forged message will send its challenge as a new message (as in the form of a reply) to the uninvolved person whose address the spammer forged as the sender of their spam creating e-mail backscatter. This effectively doubles the amount of unwanted email being distributed. Indeed, some argue that using a C/R system means sending unsolicited, bulk email (that is, spam) to all those people whose addresses are forged in spam and this can result in those systems being blacklisted on DNSBLs.
- Without the element of a recipient's consent, unsolicited emails are practically always spam. With a C/R system, however, the user is delegating their consent to the sender, which effectively renders all email as "non-spam," including those sent by spammers who fully intended to "spam" their recipients.
[edit] Social issues
Disseminating an ordinary email address that is protected by a C/R system will result in those who send mail to that address having their messages challenged unless the sender has been previously whitelisted. Many C/R critics consider it rude to give someone your email address, then require them to solve the challenge before they can send you mail.
[edit] Discrimination against the disabled
Some kinds of C/R system, such as CAPTCHAs, can discriminate against the disabled. A blind person can send and receive textual email (using a braille terminal, for instance), but cannot see an image and read text from it. A blurry image intended to defeat optical character recognition software may be impossible for sighted but visually-impaired persons.
[edit] Interactions with mailing lists
Some C/R systems interact badly with mailing list software. If a person subscribed to a mailing list begins to use C/R software, posters to the mailing list may be confronted by large numbers of challenge messages. Many regard these as junk mail equal in annoyance to actual spam. Some C/R systems allow the user to simply "whitelist" mailing lists to which they subscribe -- instructing the C/R software not to challenge their messages.
[edit] Interaction with other challenge-response systems
C/R systems can interact badly with other C/R systems. If two persons both use C/R and one emails the other, the two C/R systems will generally exhibit one of three behaviors:
- They become trapped in a loop, each challenging the other, neither one willing to deliver the challenge messages -- or the original message.
- The C/R systems attempt to avoid this by automatically whitelisting addresses to which mail is sent -- though this may not work when the recipient has their mail forwarded to a different address.
- They may recognize the challenge as "bulk" priority and elect not to challenge it, thereby avoiding a loop, but failing to deliver the message. In this case, SMTP-level rejections can ensure that the initial sender is notified of the failure by his/her own outgoing SMTP server, thereby bypassing the C/R.
[edit] Interaction with online shopping systems
Order confirmations, billing statements and delivery notices from online shopping systems are usually sent via automated systems. Since C/R systems are designed so that the challenges can not be solved by an automated system, these e-mails will usually be lost if a challenge is sent.
[edit] Challenges often look like spam or phishing e-mails
Many challenge messages look like phishing attempts since they often ask the receiver to visit an unfamiliar website or perform an equally confusing task. Other challenge messages contain advertising for the C/R system that is being used. Challenge messages that don't disclose any information about the original e-mail will often confuse the receiver of the challenge as to why they are getting it, but challenge messages that do disclose information will, in cases of forged e-mail addresses, effectively be forwarding the original spam.
[edit] Effectiveness
While some users report C/R systems are extremely effective at eliminating spam, others find that they have to review their challenged mail looking for wanted mail for which the sender has not responded to the challenge.
[edit] External links
- Challenge-Response Anti-Spam Systems Considered Harmful
- What You Need to Know About Challenge - Response Spam Filters
- John Levine: Challenge-response systems are as harmful as spam
- A Challenging Response to Challenge-Response
- Why Challenge-Response is a Bad Idea
- Challenge-Response systems make matters worse
