View source for Fetching Mail with Movemail

== Introduction ==

The '''movemail''' program in GNU Mailutils was initially designed as a better replacement for the similar utility in [http://www.gnu.org/software/emacs Emacs], whose main purpose was to move email messages from one UNIX mailbox to another.  As many other '''Mailutils''' implementations, '''movemail''' is able to do much more than its predecessor.  In fact, it offers a
great power and flexibility for a number of tasks related to electronic mail handling.

This article explains how to use '''movemail''' as a ''mail retrieval agent''. Mail retrieval agents (''MRA'' for short), are utilities that retrieve email from a remote mail server and deliver it to a local or remote email mailbox. One of the most popular ''MRAs'' is '''fetchmail'''.

== Movemail Basics ==

The '''movemail''' invocation scheme is:

<tt>
movemail [''options''] ''source-mailbox'' ''dest-mailbox''
</tt>

where ''options'' are movemail-specific command line options, ''source-mailbox'' is the mailbox to retrieve messages from, and ''dest-mailbox'' is the mailbox to move those messages to.  Both mailboxes can be specified as ''URLs'', for example:

<code>
movemail pop://user:pass@pop.example.net mh://Mail/inbox
</code>

This command will connect to ''pop.example.net'' using the POP protocol, as user ''user'' with password ''pass'', and move messages from there to the local MH folder in ''Mail/inbox''.

If the ''dest-mailbox'' URL begins with <code>smtp://</code>, '''movemail''' will re-send the retrieved messages to the SMTP server specified in the URL.  For example, to get messages from the same POP server as in the example above and to feed them to the SMTP server at localhost, one should write:

<code>
movemail pop://user:pass@pop.example.net smtp://localhost
</code>

This invocation is equivalent to the following '''fetchmail''' configuration:

<code>
 poll pop.example.net
   proto pop3
   user "user" password "pass"
   smtphost localhost
</code>

== SMTP Mailboxes ==

The SMTP URL syntax is:

<tt>
smtp://[''user''[:''pass''][;auth=''mech'',...]@]''hostname''[:''port''][;''param''[;''param''...]]
</tt>

The only mandatory part, beside the <tt>smtp://</tt> scheme designator, is ''hostname'' -- a hostname or IP address of the server to connect to. The rest of elements (shown enclosed in square brackets) is optional:

* The ''user'' part supplies the user name for authentication.
* Password for this user is supplied by the ''path'' component.
* The <tt>auth=</tt> parameter is used to specify a list of ''mechanisms'' acceptable for authentication.  Each ''mech'' is a name of a mechanism.
* ''hostname'' is the name or IP address of the host to connect to.
* ''port'' is the port number or symbolic name (from <tt>/etc/services</tt>.
* Additional parameters (''param'') are a semicolon-separated list of key-value pairs.

The ''port'' element, if supplied, specifies the port number to use instead of the default 25.  For example, to inject mail using the mail submission port (587), we can write:

<code>
movemail pop://user:pass@pop.example.net smtp://localhost:587
</code>

A service number from ''/etc/services'' can be used instead of the numeric value, as in:

<code>
movemail pop://user:pass@pop.example.net smtp://localhost:submission
</code>

The ''user'', ''pass'', and ''auth'' elements supply credentials for ESMTP authentication, if the server supports it.  See [[#TLS and ESMTP Authentication|TLS and ESMTP Authentication]], for a detailed description.

A number of additional data can be supplied using ''parameters'', shown as ''param'' in the syntax above.  These are a semicolon-separated list of words.  Each word is either a single keyword which, when present, enables a certain feature (this is called a ''boolean parameter''), or a compound statement, consisting of a ''keyword'' and a ''value'' separated with an equals sign, e.g. <tt>from=gray@gnu.org</tt>.  In the discussion below we will describe each of the available parameters where the relevant functionality is covered. For an exhaustive list of parameters, see [[#SMTP Parameters|SMTP Parameters]].

=== Mail Recipients ===

If no additional parameters are supplied, '''movemail''' tries to deduce the envelope recipient addresses, i.e. the addresses originally used in the <tt>RCPT TO</tt> commands during the SMTP transaction.  Unfortunately, there is no standard way of preserving this information in the message '''Movemail''' uses the following algorithm to deduce it.  First, the following
headers are examined:

* <tt>X-Envelope-To</tt>
* <tt>Delivered-To</tt>
* <tt>X-Original-To</tt>

The first of them which is present, is parsed and its value is used as a list of recipients.

If neither of these headers is present, '''movemail''' analyzes the value of the topmost <tt>Received</tt> header.  If it contains a ''by/for'' clause, the email following the <tt>for</tt> keyword is used as a recipient.

Finally, if everything else fails, '''movemail''' will use the values of
<tt>To</tt>, <tt>Cc</tt> and <tt>Bcc</tt> headers from the message.

Several parameters can be used to control this process.  First of all, the <tt>to</tt> parameter, if present, supplies the recipient email explicitly. Its presence turns off the above algorithm.  For example, to move all mail from the POP account of user <tt>smith</tt> to user <tt>smith@foo.com</tt> using the local SMTP server, one would write:

<code>
movemail pop://smith:pass@pop.example.net 'smtp://localhost;to=smith%40foo.com'
</code>

Notice the use of <tt>%40</tt> instead of the <tt>@</tt> character in the URL.  The ''at sign'' has a special meaning in URLs: it delimits authentication credentials from the user name, therefore it must be ''escaped'' if you want it to be treated literally.  To escape a special character in a URL, replace it with a percent sign, followed by the ASCII code of that chara
cter in hex.  Thus, the escape for <tt>:</tt> (colon) is <tt>%3A</tt>, for <tt>;</tt> is <tt>%3B</tt>, and for <tt>/</tt> is <tt>%2F</tt>.  To insert a <tt>%</tt> character verbatim, use <tt>%25</tt>.

Notice also the quotes around the last argument.  URL parameters are delimited by semicolons, which are special to the shell and therefore must be quoted or escaped.

To modify the list of headers searched for recipient addresses, use the
<tt>recipient-headers</tt> parameter.  Its value is a comma-separated list of
headers, which will be used instead of the default list.  For example:

<code>
'smtp://localhost;recipient-headers=X-Rcpt-To,X-Envelope-To,X-Original-To'
</code>

When used as a boolean (i.e. without value), <tt>recipient-headers</tt> disables header scanning:

<code>
'smtp://localhost;recipient-headers'
</code>

For any recipient address that lacks domain part, the domain part of the hostname of the local machine is appended to it.  To supply another default domain, use the <tt>domain</tt> parameter.  For example, given the following destination mailbox:

<code>
'smtp://localhost;domain=example.com'
</code>

recipient addresses without domain part will get ''@example.com'' apended to them.  Thus <tt>smith</tt> becomes <tt>smith@example.com</tt> and so on.

You can also request removing any original domain part from recipient addresses by specifying the <tt>strip-domain</tt> parameter.  When used together with <tt>domain</tt> it causes replacing of the original domain name with the new one:

<code>
'smtp://localhost;strip-domain;domain=example.com'
</code>

=== Sender Address ===

To determine the message sender '''movemail''' scans the headers of the
message looking for one of the following headers (listed in the order
of their precedence):

* <tt>X-Envelope-Sender</tt>
* <tt>X-Envelope-From</tt>
* <tt>X-Original-Sender</tt>
* <tt>From</tt>

The first header found is parsed and its value is used as sender address.
To supply a fixed sender address for all messages, use the
<tt>from</tt> parameter.

=== TLS and ESMTP Authentication ===

Unless told otherwise, '''movemail''' tries to make use of all
ESMTP capabilities offered by the remote server.  In particular,
it the SMTP server supports TLS encryption, '''movemail''' will try to
establish encrypted channel.  To disable this behavior the <tt>notls</tt>
parameter is provided.  For example, the following destination URL
forces '''movemail''' to use plaintext channel even if the server offers
a TLS capability:

<code>
'smtp://localhost;notls'
</code>

If the server supports ESMTP authentication, '''movemail''' will use it if you
supply the needed user credentials.  For example, to authenticate to
the <tt>localhost</tt> server as user <tt>smith</tt> with password
<tt>guessme</tt>, use:

<code>
smtp://smith:guessme@localhost
</code>

'''Movemail''' will select the best authentication mechanism from the list
offered by the server.  To force it to use a particular authentication
mechanism, use the <tt>auth</tt> authentication parameter.  Its value is a
comma-separated list of authentication mechanisms, in the order from
the most preferred to the least preferred one, e.g.:

<code>
'smtp://smith:guessme;auth=cram-md5,digest-md5@localhost'
</code>

When given this URL '''movemail''' will try to use either <tt>cram-md5</tt> or
<tt>digest-md5</tt>, whichever is supported by the server.  If neither of
them is supported, '''movemail''' will continue without authentication.

The <tt>auth</tt> keyword can also be used as a usual parameter, as shown
in the example below:

<code>
'smtp://smith:guessme@localhost;auth=cram-md5,digest-md5'
</code>

Putting authentication credentials in the URL itself can be unsecure,
because they become visible for other users in the <tt>ps</tt> output.  To
avoid compromising your authentication credentials, store them in
the Mailutils ''ticket file''.  See [[#Mailutils Ticket File|Mailutils Ticket File]], for a detailed description of this feature.

To turn off the ESMTP authentication, use the <tt>noauth</tt> keyword:

<code>
'smtp://localhost;noauth'
</code>

=== SMTP Parameters ===

This subsection summarizes the parameters available for use in SMTP
URLs.  For each parameter we give a short description and a reference
to the section which contains an in-depth discussion.

;auth=mech[,mech...]
: If ESMTP server supports authentication, negotiate one of the supplied
: authentication mechanisms.  See [[#TLS and ESMTP Authentication]].

;domain=''string''
: Use ''string'' as a domain part for those recipients that lack it.
See [[#Mail Recipients|Mail Recipients]].

;from=''addr''
:Use ''addr'' as sender address.  See [[#Sender Address|Sender Address]].

;noauth
:Do not use ESMTP authentication.  See [[#TLS and ESMTP Authentication]].

;notls
:Do not use TLS, even if the SMTP server supports it.  See [[#TLS and ESMTP Authentication]].

;recipient-headers[=name[,name...]]
:Use the supplied header names to determine recipient addresses.
:See [[#Mail Recipients|Mail Recipients]].  When no values are supplied, disables header scanning.

;strip-domain
:Strip domain part from all recipient addresses. See [[#Mail Recipients|Mail Recipients]].

;to=addr[,addr]
:Deliver messages to the supplied addresses.  See [[#Mail Recipients|Mail Recipients]].

== Mailutils Ticket File ==

If, when trying to authenticate to a remote server, '''movemail''' sees that
the corresponding URL does not provide required credentials, it will
try to obtain them from a ''ticket file''.

''Ticket file'' is a plain text file, located in the home
directory of the current user, and named ''.mu-tickets''.  It
is formatted as a usual UNIX line-oriented configuration
file, i.e. comments are introduced by a <tt>#</tt> sign and extend to the end
of the line on which this sign appears, emtpy lines and comments are
ignored.  Each remaining line must be formatted as a URL, which
contains at least one of ''username'' and ''password'' parts.  We call
such an URL a ''ticket''.

When scanning this file, '''movemail''' looks for a ticket that matches
the actual URL most closely.  The ''username'' and ''password'' from that
ticket are then used for authentication.
The following URL parts are considered when matching: ''scheme'',
''user'', ''host'' and ''port''.  If a part consists
of a single ''wildcard'' (<tt>*</tt>), it matches any value in the
corresponding part of the actual URL.  Otherwise the parts are
compared as follows:

* ''scheme'' and ''host'' use case-insensitve comparison.
* ''user'' uses case-sensitive comparison.
* ''port'' is compared for numeric equality.

A missing part in the ticket entry is treated exactly as if a wildcard
were present instead.

For example, the URL <tt>smtp://gnu.org</tt> matches the following ticket entries:

<code>
 smtp://foo:bar@*
 smtp://bar:baz@gnu.org
 *://baz:qux@*
 *://quux:bar@gnu.org
</code>

If several tickets match the URL, '''movemail''' selects the best match by
using the following algorithm.  Each matching entry is assigned a
''penalty value'' computed as a sum of individual penalties for each
part of the URL.  The individual penalty is 0 if the ticket part
equals the corresponding URL part.  Otherwise, if this part is missing
either from the ticket or from the URL, or if the ticket part is
represented by a wildcard, the penalty value is 4 for ''user'' part, 3
for ''scheme'' part, 2 for ''host'' part and 1 for ''port'' part.  Once
penalty values are computed, '''movemail''' selects the matching ticket
with the lowest penalty value.

In the example above, the best match is <tt>smtp://bar:baz@gnu.org</tt>.

{{note|Use ''mu wicket'' command to check which ticket will be matched by
a particular URL}}

If some of the credentials is still missing after consulting the
ticket file, '''movemail''' will ask the user for the missing credential
(this is possible, of course, only if the controlling terminal is a
<tt>tty</tt>).

Consider this example:

<code>
 $ movemail pop://example.net smtp://gray@localhost
 Pop user: gray  ''# The user types in his username''
 Pop password:   ''# The user types the password for his POP account at <tt>example.net</tt>''
 SMTP password:  ''# The user types the password for SMTP at <tt>localhost</tt>''
</code>

{{note|input passwords are not echoed.}}

== POP3 Mailboxes ==

The syntax for POP3 URLs is similar to SMTP:

<code>
pop://[''user''[:''pass''][;auth=+APOP]@]''hostname''[:''port''][;notls]
</code>

The ''hostname'' gives the name or IP address of the host running a POP3
server.  Optional ''port'' is used to connect to a port other than the
default 110.

The ''user'' and ''pass'' supply authentication credentials.  If any of
them is missing, '''movemail''' first tries to obtain it from the
[[#Mailutils Ticket File|ticket file]].  It that fails,
its behavior depends on the type of the controlling terminal.  If the
terminal is a <tt>tty</tt> device (i.e. the program was started from the
command line), it will ask the user to supply the missing credentials.
Otherwise it will issue an appropriate error message and terminate.

By default, the usual POP3 authentication is used.  The <tt>auth=+APOP</tt>
authentication parameter instructs '''movemail''' to use APOP
authentication instead.

If the server offers the STLS capability, '''movemail''' attempts to
establish encrypted TLS connection.  To disable this behavior use
the <tt>notls</tt> parameter.

The deprecated <tt>pops</tt> protocol is also supported.  It is enabled by
an URL beginning with <tt>pops://</tt> instead of <tt>pop://</tt>.
The URL syntax is similar to the above:

<code>
pops://[''user''[:''pass''][;auth=+APOP]@]''hostname''[:''port'']
</code>

Unless ''port'' is given, the default value 995 is used.  The <tt>notls</tt>
parameter is not supported, since POPS goes over an already encrypted
channel.

For example, to fetch mail from your '''gmail''' account and submit it to
the local MSA, use:

<code>
movemail pops://''rname'':''pass''@pop.gmail.com 'smtp://localhost:587;to=''lname'''
</code>

Replace ''rname'' and ''pass'' with your user name and password on '''gmail'''
and ''lname'' with your local name.