Variable envelope return path
From Wikipedia, the free encyclopedia
Variable envelope return path (VERP) is a technique used by some electronic mailing list software to enable automatic detection and removal of undeliverable e-mail addresses. It works by using a different return path (also called "envelope sender") for each recipient of a message.
Contents |
[edit] Motivation
Any long-lived mailing list is going to eventually contain addresses that can't be reached. Addresses that were once valid can become unusable because the person receiving the mail there has switched to a different provider (possibly as a result of changing jobs or schools). In another scenario, the address may still exist but be abandoned, with unread mail accumulating until there is not enough room left to accept any more.
When a message is sent to a mailing list, the mailing list software re-sends it to all of the addresses on the list. The presence of invalid addresses in the list results in bounce messages being sent to the owner of the list. If the mailing list is small, the owner can read the bounce messages and manually remove the invalid addresses from the list. With a larger mailing list, this is a tedious, unpleasant job, so it is desirable to automate the process.
Unfortunately, most bounce messages have historically been designed to be read by human users, not automatically handled by software. They all convey the same basic idea (the message from X to Y could not be delivered because of reason Z) but with so many variations that it would be nearly impossible to write a program to reliably interpret the meaning of every bounce message. RFC 1894 (obsoleted by RFC 3464) defines a standard format to fix this problem, but support for the standard is far from universal. Having said that, there are several common formats RFC 3464, qmail's qsbmf and Microsoft's DSN format for Exchange that cover large proportion of bounces.
Microsoft Exchange can sometimes bounce a message without any indication of the address the original message was sent to! When Exchange knows of the intended recipient, but is not willing to accept email for them, it omits their address. If a message is sent to joe@example.com and the server knows that this is Joe User, it will bounce the message saying that the message to Joe User could not be delivered, leaving out the joe@example.com address altogether. VERP is the only viable way of being able to handle such bounces correctly.
[edit] How VERP solves the bounce handling problem
The hard part of bounce handling is matching up a bounce message with the undeliverable address which caused the bounce. If the mailing list software can see that a bounce resulted from an attempt to send a message to user@example.com then it doesn't need to understand the rest of the information in the bounce. It can simply count how many messages were recently sent to user@example.com, and how many bounces resulted, and if the proportion of bounced messages is too high, the address is removed from the list.
While bounce message formats in general vary wildly, there is one aspect of a bounce message that is highly predictable: the address to which it will be sent. VERP takes full advantage of this. In a mailing list that uses VERP, a different sender address is used for each recipient.
The mailing list manager knows that it sent a message from X to Y, so if a bounce message is received at address X, it can only be because address Y was undeliverable, because nothing was sent from X to any other destination. Thus the important information has been extracted from the bounce message, without any need to understand its contents, which means the person in charge of the list does not need to deal with it manually.
[edit] Origin
The first serious advocate of this solution, and the originator of the term VERP to describe it, was Daniel J. Bernstein, who first put the idea into practice in his qmail MTA and ezmlm mailing list manager. See his original explanation, dated February 1, 1997.
[edit] Example
Assume there is mailing list called wikipedians@example.net and that an individual, bob@example.org has subscribed to it, but later on, Bob has left example.org, so his address is no longer valid. Consider what happens when someone sends a message to the list.
[edit] Without VERP
Without VERP, the mailing list manager might send a message with the following characteristics:
- envelope sender:
wikipedians-owner@example.net - recipient:
bob@example.org
This would result in a bounce, generated by the MTA of either example.net or example.org, with the following characteristics:
- envelope sender: empty
- recipient:
wikipedians-owner@example.net - contents: example.org was unable to deliver the following message to bob: ...
The mailing list manager can't be expected to understand the contents of this bounce, and can't deduce anything from the recipient address because hundreds of other people besides Bob were also sent messages from wikipedians-owner@example.net.
[edit] With VERP
With VERP, the original message would be different:
- envelope sender:
wikipedians-owner-bob=example.org@example.net - recipient: <bob@example.org>
The bounce, then, will be more useful:
- envelope sender: empty
- recipient:
wikipedians-owner-bob=example.org@example.net - contents: example.org was unable to deliver the following message to bob: ...
From this bounce message the mailing list manager can deduce that a message to bob@example.org must have failed.
This example shows the simplest possible method of matching a VERP to a list subscriber: the entire recipient address is included within the return path, with the at sign replaced by an equals sign because a return path with two at signs would be invalid. Other encoding schemes are possible.
[edit] Software that supports VERP
- ezmlm
- GNU Mailman
- postfix
- qmail
- Sendmail, with a ruleset.
- Mercury Mail Transport System
[edit] Disadvantages
The use of VERP requires each message to be sent once for every recipient, instead of once to each receiving SMTP server. This is because of a limitation of SMTP, which allows multiple recipient addresses to be specified in a single transaction, but only one sender address. When there are many subscribers in the same domain, a mailing list that is not using VERP can combine multiple deliveries into a single transaction. It connects to the appropriate server for the domain, gives the single sender address, the recipient addresses, and then sends the message contents only once.
A mailing list using VERP, on the other hand, must send the entire message body repeatedly, which leads to an overall increase in bandwidth usage. This inefficiency is usually not considered a big problem, especially by qmail users, since qmail always sends messages once per recipient, even when VERP is not being used. Some packages mitigate the impact of VERP by applying it selectively, for example a mailing list manager might only use VERP on 1 in 10 mailings. This way you can gain much of VERP's tight bounce control and accurate feedback without incurring the processing and network overhead every time.
Another problem with VERP (and with any automatic bounce handling scheme) is that there will always be some recalcitrant MTAs on the Internet that fail to follow even the simplest standards. VERP depends on the recipients' MTAs following the rule that bounces are sent to the envelope sender. This has been a standard requirement since the dawn of SMTP in 1982 (see RFC 821), but still there are MTAs that get it wrong, usually by bouncing to the address in the From: header (See Imail for an example).
Systems that implement greylisting will cause each post to the mailing list to be delayed unless they treat "similar" sender addresses as being equivalent.
