Fetching Mail with Movemail

Introduction

The movemail program in GNU Mailutils was initially designed as a better replacement for the similar utility in Emacs, whose main purpose was to move email messages from one UNIX mailbox to another. As many other Mailutils implementations, movemail is able to do much more than its predecessor. In fact, it offers a great power and flexibility for a number of tasks related to electronic mail handling.

This article explains how to use movemail as a mail retrieval agent. Mail retrieval agents (MRA for short), are utilities that retrieve email from a remote mail server and deliver it to a local or remote email mailbox. One of the most popular MRAs is fetchmail.

Movemail Basics

The movemail invocation scheme is:

 movemail [options] source-mailbox dest-mailbox

where options are movemail-specific command line options, source-mailbox is the mailbox to retrieve messages from, and dest-mailbox is the mailbox to move those messages to. Both mailboxes can be specified as URLs, for example:

 movemail pop://user:pass@pop.example.net mh://Mail/inbox

This command will connect to pop.example.net using the POP protocol, as user user with password pass, and move messages from there to the local MH folder in Mail/inbox.

If the dest-mailbox URL begins with smtp://, movemail will re-send the retrieved messages to the SMTP server specified in the URL. For example, to get messages from the same POP server as in the example above and to feed them to the SMTP server at localhost, one should write:

 movemail pop://user:pass@pop.example.net smtp://localhost

This invocation is equivalent to the following fetchmail configuration:

poll pop.example.net
  proto pop3
  user "user" password "pass"
  smtphost localhost

SMTP Mailboxes

The SMTP URL syntax is:

 smtp://[user[:pass][;auth=mech,...]@]hostname[:port][;param[;param...]]

The only mandatory part, beside the smtp:// scheme designator, is hostname -- a hostname or IP address of the server to connect to. The rest of elements (shown enclosed in square brackets) is optional:

• The user part supplies the user name for authentication.
• Password for this user is supplied by the path component.
• The auth= parameter is used to specify a list of mechanisms acceptable for authentication. Each mech is a name of a mechanism.
• hostname is the name or IP address of the host to connect to.
• port is the port number or symbolic name (from /etc/services.
• Additional parameters (param) are a semicolon-separated list of key-value pairs.

The port element, if supplied, specifies the port number to use instead of the default 25. For example, to inject mail using the mail submission port (587), we can write:

 movemail pop://user:pass@pop.example.net smtp://localhost:587

A service number from /etc/services can be used instead of the numeric value, as in:

 movemail pop://user:pass@pop.example.net smtp://localhost:submission

The user, pass, and auth elements supply credentials for ESMTP authentication, if the server supports it. See TLS and ESMTP Authentication, for a detailed description.

A number of additional data can be supplied using parameters, shown as param in the syntax above. These are a semicolon-separated list of words. Each word is either a single keyword which, when present, enables a certain feature (this is called a boolean parameter), or a compound statement, consisting of a keyword and a value separated with an equals sign, e.g. from=gray@gnu.org. In the discussion below we will describe each of the available parameters where the relevant functionality is covered. For an exhaustive list of parameters, see SMTP Parameters.

Mail Recipients

If no additional parameters are supplied, movemail tries to deduce the envelope recipient addresses, i.e. the addresses originally used in the RCPT TO commands during the SMTP transaction. Unfortunately, there is no standard way of preserving this information in the message Movemail uses the following algorithm to deduce it. First, the following headers are examined:

• X-Envelope-To
• Delivered-To
• X-Original-To

The first of them which is present, is parsed and its value is used as a list of recipients.

If neither of these headers is present, movemail analyzes the value of the topmost Received header. If it contains a by/for clause, the email following the for keyword is used as a recipient.

Finally, if everything else fails, movemail will use the values of To, Cc and Bcc headers from the message.

Several parameters can be used to control this process. First of all, the to parameter, if present, supplies the recipient email explicitly. Its presence turns off the above algorithm. For example, to move all mail from the POP account of user smith to user smith@foo.com using the local SMTP server, one would write:

 movemail pop://smith:pass@pop.example.net 'smtp://localhost;to=smith%40foo.com'

Notice the use of %40 instead of the @ character in the URL. The at sign has a special meaning in URLs: it delimits authentication credentials from the user name, therefore it must be escaped if you want it to be treated literally. To escape a special character in a URL, replace it with a percent sign, followed by the ASCII code of that chara cter in hex. Thus, the escape for : (colon) is %3A, for ; is %3B, and for / is %2F. To insert a % character verbatim, use %25.

Notice also the quotes around the last argument. URL parameters are delimited by semicolons, which are special to the shell and therefore must be quoted or escaped.

To modify the list of headers searched for recipient addresses, use the recipient-headers parameter. Its value is a comma-separated list of headers, which will be used instead of the default list. For example:

 'smtp://localhost;recipient-headers=X-Rcpt-To,X-Envelope-To,X-Original-To'

When used as a boolean (i.e. without value), recipient-headers disables header scanning:

 'smtp://localhost;recipient-headers'

For any recipient address that lacks domain part, the domain part of the hostname of the local machine is appended to it. To supply another default domain, use the domain parameter. For example, given the following destination mailbox:

 'smtp://localhost;domain=example.com'

recipient addresses without domain part will get @example.com appended to them. Thus smith becomes smith@example.com and so on.

You can also request removing any original domain part from recipient addresses by specifying the strip-domain parameter. When used together with domain it causes replacing of the original domain name with the new one:

 'smtp://localhost;strip-domain;domain=example.com'

Sender Address

To determine the message sender movemail scans the headers of the message looking for one of the following headers (listed in the order of their precedence):

• X-Envelope-Sender
• X-Envelope-From
• X-Original-Sender
• From

The first header found is parsed and its value is used as sender address. To supply a fixed sender address for all messages, use the from parameter.

TLS and ESMTP Authentication

Unless told otherwise, movemail tries to make use of all ESMTP capabilities offered by the remote server. In particular, it the SMTP server supports TLS encryption, movemail will try to establish encrypted channel. To disable this behavior the notls parameter is provided. For example, the following destination URL forces movemail to use plaintext channel even if the server offers a TLS capability:

 'smtp://localhost;notls'

If the server supports ESMTP authentication, movemail will use it if you supply the needed user credentials. For example, to authenticate to the localhost server as user smith with password guessme, use:

 smtp://smith:guessme@localhost

Movemail will select the best authentication mechanism from the list offered by the server. To force it to use a particular authentication mechanism, use the auth authentication parameter. Its value is a comma-separated list of authentication mechanisms, in the order from the most preferred to the least preferred one, e.g.:

 'smtp://smith:guessme;auth=cram-md5,digest-md5@localhost'

When given this URL movemail will try to use either cram-md5 or digest-md5, whichever is supported by the server. If neither of them is supported, movemail will continue without authentication.

The auth keyword can also be used as a usual parameter, as shown in the example below:

 'smtp://smith:guessme@localhost;auth=cram-md5,digest-md5'

Putting authentication credentials in the URL itself hurts security, because they become visible for other users in the ps output. To avoid compromising your authentication credentials, store them in the Mailutils ticket file. See Mailutils Ticket File, for a detailed description of this feature.

To turn off the ESMTP authentication, use the noauth keyword:

 'smtp://localhost;noauth'

SMTP Parameters

This subsection summarizes the parameters available for use in SMTP URLs. For each parameter we give a short description and a reference to the section which contains an in-depth discussion.

auth=mech[,mech...]

If ESMTP server supports authentication, negotiate one of the supplied

authentication mechanisms. See #TLS and ESMTP Authentication.

domain=string

Use string as a domain part for those recipients that lack it.

See Mail Recipients.

from=addr

Use addr as sender address. See Sender Address.

noauth

Do not use ESMTP authentication. See #TLS and ESMTP Authentication.

notls

Do not use TLS, even if the SMTP server supports it. See #TLS and ESMTP Authentication.

recipient-headers[=name[,name...]]

Use the supplied header names to determine recipient addresses.

See Mail Recipients. When no values are supplied, disables header scanning.

strip-domain

Strip domain part from all recipient addresses. See Mail Recipients.

to=addr[,addr]

Deliver messages to the supplied addresses. See Mail Recipients.

Mailutils Ticket File

If, when trying to authenticate to a remote server, movemail sees that the corresponding URL does not provide required credentials, it will try to obtain them from a ticket file.

Ticket file is a plain text file, located in the home directory of the current user, and named .mu-tickets. It is formatted as a usual UNIX line-oriented configuration file, i.e. comments are introduced by a # sign and extend to the end of the line on which this sign appears, empty lines and comments are ignored. Each remaining line must be formatted as a URL, which contains at least one of username and password parts. We call such an URL a ticket.

When scanning this file, movemail looks for a ticket that matches the actual URL most closely. The username and password from that ticket are then used for authentication. The following URL parts are considered when matching: scheme, user, host and port. If a part consists of a single wildcard (*), it matches any value in the corresponding part of the actual URL. Otherwise the parts are compared as follows:

• scheme and host use case-insensitive comparison.
• user uses case-sensitive comparison.
• port is compared for numeric equality.

A missing part in the ticket entry is treated exactly as if a wildcard were present instead.

For example, the URL smtp://gnu.org matches the following ticket entries:

smtp://foo:bar@*
smtp://bar:baz@gnu.org
*://baz:qux@*
*://quux:bar@gnu.org

If several tickets match the URL, movemail selects the best match by using the following algorithm. Each matching entry is assigned a penalty value computed as a sum of individual penalties for each part of the URL. The individual penalty is 0 if the ticket part equals the corresponding URL part. Otherwise, if this part is missing either from the ticket or from the URL, or if the ticket part is represented by a wildcard, the penalty value is 4 for user part, 3 for scheme part, 2 for host part and 1 for port part. Once penalty values are computed, movemail selects the matching ticket with the lowest penalty value.

In the example above, the best match is smtp://bar:baz@gnu.org.

Note: Use the mu wicket command to check which ticket will be matched by a particular URL.

If some of the credentials is still missing after consulting the ticket file, movemail will ask the user for the missing credential (this is possible, of course, only if the controlling terminal is a tty).

Consider this example:

$ movemail pop://example.net smtp://gray@localhost
Pop user: gray  # The user types in his username
Pop password:   # The user types the password for his POP account at example.net
SMTP password:  # The user types the password for SMTP at localhost

Note: input passwords are not echoed.

POP3 Mailboxes

The syntax for POP3 URLs is similar to SMTP:

 pop://[user[:pass][;auth=+APOP]@]hostname[:port][;notls]

The hostname gives the name or IP address of the host running a POP3 server. Optional port is used to connect to a port other than the default 110.

The user and pass supply authentication credentials. If any of them is missing, movemail first tries to obtain it from the ticket file. It that fails, its behavior depends on the type of the controlling terminal. If the terminal is a tty device (i.e. the program was started from the command line), it will ask the user to supply the missing credentials. Otherwise it will issue an appropriate error message and terminate.

By default, the usual POP3 authentication is used. The auth=+APOP authentication parameter instructs movemail to use APOP authentication instead.

If the server offers the STLS capability, movemail attempts to establish encrypted TLS connection. To disable this behavior, use the notls parameter.

POP3S

The deprecated POP3S protocol is also supported. It is enabled by an URL beginning with pops:// instead of pop://. The URL syntax is similar to the above:

 pops://[user[:pass][;auth=+APOP]@]hostname[:port]

Unless port is given, the default value 995 is used. The notls parameter is not supported, since POP3S goes over an already encrypted channel.

For example, to fetch mail from your gmail account and submit it to the local MSA, use:

 movemail pops://rname:pass@pop.gmail.com 'smtp://localhost:587;to=lname'

Replace rname and pass with your user name and password on gmail and lname with your local name.

Error Handling

Various errors can occur during retrieval of each particular messages as well as during injecting it into the MTA queue. When an error occurs, movemail normally stops any further processing and terminates, without deleting already moved messages. Although such a behavior is a safest way, there may be cases when it is not desired. Therefore, a special configuration option is provided, which controls the movemail actions when an error occurs while trying to append the message to the destination mailbox:

 onerror keyword[,keyword...];

A keyword is one of the following:

abort

Abort any further processing, reset source mailbox to its original state, and terminate with code 1. This is the default behavior. No other keyword can be given together with abort.

skip

Skip this message and try to move the next one. None of the messages is deleted before closing the mailbox. Movemail terminates with code 1.

count

Count this message as successfully processed, but do not mark it for deletion. Movemail expunges the successfully processed messages from the source mailbox and terminates with code 0.

delete

Mark the message for deletion, and continue processing. Before exiting, movemail will expunge both successfully copied messages and failed ones. It will, however, terminate with code 1, unless count is also given.

These keywords can also be given from the command line, using the --onerror command line option.