Sender Policy Framework

Sender Policy Framework (SPF) is a simple email-validation system designed to detect email spoofing by providing a mechanism to allow receiving mail exchangers to check that incoming mail from a domain comes from a host authorized by that domain's administrators.^[1] The list of authorized sending hosts for a domain is published in the Domain Name System (DNS) records for that domain in the form of a specially formatted TXT record. Email spam and phishing often use forged "from" addresses, so publishing and checking SPF records can be considered anti-spam techniques.

IETF publication RFC 7208 dated April 2014 defines Sender Policy Framework as "proposed standard".

History

The first public mention of the concept was in 2000 but went mostly unnoticed.^[2] No mention was made of the concept again until a first attempt at an SPF-like specification was published in 2002 on the IETF "namedroppers" mailing list by Dana Valerie Reese (née Green), who was unaware of the 2000 mention of the idea. The very next day, Paul Vixie posted his own SPF-like specification on the same list. These posts ignited a lot of interest, and eventually led to the forming of the IETF Anti-Spam Research Group (ASRG) and their mailing list, where the SPF idea was debated among a subscriber base that seemed to grow exponentially day by day. Among the proposals submitted to the ASRG were "Reverse MX" by Hadmut Danisch, and "Designated Mailer Protocol" by Gordon Fecyk.^[3]

In June 2003, Meng Weng Wong merged the RMX and DMP specifications^[4] and solicited suggestions from other programmers. Over the next six months, a large number of changes were made and a large community had started working on SPF.^[5]

Originally SPF stood for Sender Permitted From and was sometimes also called SMTP+SPF, but its name was changed to Sender Policy Framework in February 2004.

In early 2004, the IETF created the MARID working group and tried to use SPF and Microsoft's CallerID proposal as the basis for what is now known as Sender ID.

After the collapse of MARID, the SPF community returned to the original "classic" version of SPF. In July 2005, this version of the specification was approved by the IESG as an IETF experiment, inviting the community to observe SPF during the two years following publication. On April 28, 2006, the SPF RFC was published as experimental RFC 4408.

In April 2014 IETF publish SPF in RFC 7208 as "proposed standard".

Principles of operation

The Simple Mail Transfer Protocol permits any computer to send email claiming to be from any source address. This is exploited by spammers who often use forged email addresses,^[6] making it more difficult to trace a message back to its source, and easy for spammers to hide their identity in order to avoid responsibility. It is also used in phishing techniques, where users can be duped into disclosing private information in response to an email purportedly sent by an organization such as a bank.

SPF allows the owner of an Internet domain to specify which computers are authorized to send mail with "from" addresses in that domain, using Domain Name System (DNS) records. Receivers verifying the SPF information in TXT records may reject messages from unauthorized sources before receiving the body of the message. Thus, the principles of operation are similar to those of DNS-based blackhole lists (DNSBL), except that SPF uses the authority delegation scheme of the Domain Name System. Current practice requires the use of TXT records,^[7] just as early implementations did. For a while a new record type (SPF, type 99) was registered and made available in common DNS software. Use of TXT records for SPF was intended as a transitional mechanism at the time. The experimental RFC, RFC 4408, section 3.1.1, suggested "an SPF-compliant domain name SHOULD have SPF records of both RR types".^[8] The proposed standard, RFC 7208, says "use of alternative DNS RR types was supported in SPF's experimental phase but has been discontinued".^[7]

The "from" address is transmitted at the beginning of the SMTP dialog. If the server rejects the domain, the unauthorized client should receive a rejection message, and if that client was a relaying message transfer agent (MTA), a bounce message to the original from address may be generated. If the server accepts the domain, and subsequently also accepts the recipients and the body of the message, it should insert a Return-Path field in the message header in order to save the from address. While the address in the Return-Path often matches other originator addresses in the mail header such as from, this is not necessarily the case, and SPF does not prevent forgery of these other addresses such as sender.

Spammers can send email with an SPF PASS result if they have an account in a domain with a sender policy, or abuse a compromised system in this domain. However, doing so makes the spammer easier to trace.

The main benefit of SPF is to the owners of e-mail addresses that are forged in the Return-Path. They receive large numbers of unsolicited error messages and other auto-replies. If such receivers use SPF to specify their legitimate source IP addresses and indicate FAIL result for all other addresses, receivers checking SPF can reject forgeries, thus reducing or eliminating the amount of backscatter.

SPF has potential advantages beyond helping identify unwanted mail. In particular, if a sender provides SPF information, then receivers can use SPF PASS results in combination with a white list to identify known reliable senders. Scenarios like compromised systems and shared sending mailers limit this use.

Reasons to implement

If a domain publishes an SPF record, spammers and phishers are less likely to forge e-mails pretending to be from that domain, because the forged e-mails are more likely to be caught in spam filters which check the SPF record. Therefore, an SPF-protected domain is less attractive to spammers and phishers. Because an SPF-protected domain is less attractive as a spoofed address, it is less likely to be blacklisted by spam filters and so ultimately the legitimate e-mail from the domain is more likely to get through.^[9]

FAIL and forwarding

SPF breaks plain message forwarding. When a domain publishes an SPF FAIL policy, legitimate messages sent to receivers forwarding their mail to third parties may be rejected and/or bounced if all of the following occur:

1. The forwarder does not rewrite the Return-Path, unlike mailing lists.
2. The next hop does not whitelist the forwarder.
3. This hop checks SPF.

This is a necessary and obvious feature of SPF – checks behind the "border" MTA (MX) of the receiver cannot work directly.

Publishers of SPF FAIL policies must accept the risk that their legitimate email are being rejected or bounced. They should test (e.g., with a SOFTFAIL policy) until they are satisfied with the results. See below for a list of alternatives to plain message forwarding.

HELO tests

For an empty Return-Path as used in error messages and other auto-replies, an SPF check of the HELO-identity is mandatory.

With a bogus HELO identity the result NONE would not help, but for valid host names SPF also protects the HELO identity. This SPF feature was always supported as an option for receivers, and later SPF drafts including the final specification recommend to check the HELO always.

This allows receivers to white list sending mailers based on a HELO PASS, or to reject all mails after a HELO FAIL. It can also be used in reputation systems (any white or black list is a simple case of a reputation system).

Implementation

Compliance with SPF consists of three loosely related tasks:

• Publish a policy: Domains and hosts identify the machines authorized to send e-mail on their behalf. They do this by adding additional records to their existing DNS information: every domain name or host that has an A record or MX record should have an SPF record specifying the policy if it is used either in an email address or as HELO/EHLO argument. Hosts which do not send mail should have an SPF record published which indicate such ("v=spf1 -all"). It is highly recommended to validate the SPF record using record testing tools such as those provided on the SPF Project webpage.
• Check and use SPF information: Receivers use ordinary DNS queries, which are typically cached to enhance performance. Receivers then interpret the SPF information as specified and act upon the result.
• Revise mail forwarding: Plain mail forwarding is not allowed by SPF. The alternatives are:
  • Remailing (i.e., replacing the original sender with one belonging to the local domain)
  • Refusing (i.e., answering 551 User not local; please try <user@example.com>)
  • Whitelisting on the target server, so that it will not refuse a forwarded message
  • Sender Rewriting Scheme, a more complicated mechanism that handles routing non-delivery notifications to the original sender

Thus, the key issue in SPF is the specification for the new DNS information that domains set and receivers use. The records laid out below are in typical DNS syntax:

example.com. IN TXT "v=spf1 ip4:192.0.2.0/24 ip4:198.51.100.123 a -all"

"v=" defines the version of SPF used. The following words provide mechanisms to use to determine if a domain is eligible to send mail. The "ip4" and "a" specify the systems permitted to send messages for the given domain. The "-all" at the end specifies that, if the previous mechanisms did not match, the message should be rejected.

Mechanisms

Eight mechanisms are defined:

Qualifiers

Each mechanism can be combined with one of four qualifiers:

• + for a PASS result. This can be omitted; e.g., +mx is the same as mx.
• ? for a NEUTRAL result interpreted like NONE (no policy).
• ~ (tilde) for SOFTFAIL, a debugging aid between NEUTRAL and FAIL. Typically, messages that return a SOFTFAIL are accepted but tagged.
• - (minus) for FAIL, the mail should be rejected (see below).

Modifiers

The modifiers allow for future extensions to the framework. To date only the two modifiers defined in the RFC 4408 have been widely deployed:

• exp=some.example.com gives the name of a domain with a DNS TXT record (interpreted using SPF's macro language) to get an explanation for FAIL results—typically a URL which is added to the SMTP error code. This feature is rarely used.
• redirect=some.example.com can be used instead of the ALL-mechanism to link to the policy record of another domain. This modifier is easier to understand than the somewhat similar INCLUDE-mechanism.

Error handling

As soon as SPF implementations detect syntax errors in a sender policy they must abort the evaluation with result PERMERROR. Skipping erroneous mechanisms cannot work as expected, therefore include:bad.example and redirect=bad.example also cause a PERMERROR.

Another safeguard is the maximum of ten mechanisms querying DNS, i.e. any mechanism except from IP4, IP6, and ALL. Implementations can abort the evaluation with result SOFTERROR when it takes too long or a DNS query times out, but they must return PERMERROR if the policy directly or indirectly needs more than ten queries for mechanisms. Any redirect= also counts towards this processing limit.

A typical SPF HELO policy v=spf1 a -all may execute up to three DNS queries: (1) TXT, (2) SPF (obsoleted by RFC 7208), and (3) A or AAAA. This last query counts as the first mechanism towards the limit (10). In this example it is also the last, because ALL needs no DNS lookup.

Controversy

In 2004, Steven M. Bellovin wrote an e-mail discussing his concerns with SPF.^[10] Some of these include:

• SPF originally used TXT records in DNS, which are supposed to be free-form text with no semantics attached. SPF proponents readily acknowledge that it would be better to have records specifically designated for SPF, but this choice was made to enable rapid implementation of SPF. In July 2005, IANA assigned the Resource Record type 99 to SPF. Later on,^[7] the use of SPF records was discontinued, and as of 2017, it is still necessary to use TXT records.^[7]
• At the time he wrote his message there was no consensus that SPF was the right way to go. Some major e-mail service providers have not bought into this scheme. Unless and until they do, it does not help much, either for their customers (who make up a substantial proportion of the user population) or for everyone else (because their addresses could be forged). It is worth noting that since this concern was raised, Google Mail and AOL, among others, have embraced SPF.^[11]
• Bellovin's strongest concerns involve the underlying assumptions of SPF (its "semantic model"). When using SPF, the SPF DNS records determine how a sender is allowed to send, meaning that the owner of the domain will control how senders are allowed to send. People who use "portable" e-mail addresses (such as e-mail addresses created by professional organizations) will be required to use the domain owner's SMTP sender, which may not even exist. Organizations providing these "portable" addresses could, however, create their own mail submission agents (MSAs) (RFC 6409) or offer VPNs or simply not publish an SPF record. Additionally, SPF only ties the SMTP Return-Path to permitted MSAs; users are still free to use their RFC 5322 addresses elsewhere.

There are other concerns about the impact of widespread use of SPF, notably the impact on various legitimate forms of email spoofing,^[12] such as forwarding services, SMTP use by people with multiple identities, etc. (For example, a person who uses their home ISP's SMTP servers to send mail with their work email as the address.) On the other hand, many of these uses may be "expected" yet not "legitimate". To a certain extent this is more a question of ownership and expectations than a technical question.

The IETF spfbis working group, tasked with reworking the SPF specification aiming for "Proposed Standard" status in a new RFC, during April 2013 appeared to have reached consensus around deprecating SPF type 99 in favour of continued TXT record usage.^[13] People from the DNSEXT working group strongly opposed this in a series of email threads on spfbis, dnsext, and IETF general discussion mailing lists.^[14][6] The spfbis working group chair requested an end to that torrent of protest, since the discussion on the resource record type (RRTYPE) in the working group was terminated long ago,^[15] a move that was seen as trying to silence the protests by some fierce DNS purists. An independent draft was proposed later, documenting how the spurious recursion to TXT records is characterized in the current Internet.^[16]

Deployment

Anti-spam software such as SpamAssassin version 3.0.0 and ASSP implement SPF. Many mail transfer agents (MTAs) support SPF directly such as Courier, CommuniGate Pro, Wildcat, MDaemon, and Microsoft Exchange, or have patches or plug-ins available that support SPF, including Postfix, Sendmail, Exim, qmail, and Qpsmtpd.^[17] As of 2013, more than seven million domains publish SPF FAIL -all policies.^[18]

In August 2005 it was learned that EarthLink would refuse to allow hosted domains the ability to enter SPF records.^[19]

In a survey published in 2007, 5% of the .com and .net domains had some kind of SPF policy. In 2009, a continuous survey run at Nokia Research reports that 51% of the tested domains specify an SPF policy.^[20] These results can include trivial policies like v=spf1 ?all.^[21] In April 2007, BITS, a division of the Financial Services Roundtable, published e-mail security recommendations for its members including SPF deployment.^[22]

In 2008, the Messaging Anti-Abuse Working Group (MAAWG) published a paper about email authentication covering SPF, Sender ID, and DomainKeys Identified Mail (DKIM).^[23] In their "Sender Best Communication Practices" the MAAWG stated: "At the very least, senders should incorporate SPF records for their mailing domains".^[24]

In 2015, the Messaging Anti-Abuse Working Group (MAAWG) revised a paper about email authentication covering SPF, DomainKeys Identified Mail (DKIM), and DMARC (DMARC). In their revised "Sender Best Communication Practices" the MAAWG stated: "Authentication supports transparency by further identifying the sender(s) of a message, while also contributing to the reduction or elimination of spoofed and forged addresses".^[25]

See also

• Author Domain Signing Practices
• DMARC

References

External links

• IETF RFC4408: Sender Policy Framework (SPF) for Authorizing Use of Domains in E-Mail, Version 1 EXPERIMENTAL (2006)
• IETF RFC6652: Sender Policy Framework (SPF) Authentication Failure Reporting Using the Abuse Reporting Format, PROPOSED STANDARD (2012)
• libspf2 – An Open Source Implementation of the SPF Protocol (2010)
• Unified eMail: Free SPF Analyzer (2015)