F11 fetchmail

F11 fetchmail Configuration

fetchmail allows you to efficiently fetch mail from a remote server to a local e-mail box, while providing great flexibility in the process. fetchmail is capable of retrieving mail from POP, IMAP, ETRN, or an ODMR e-mail servers.

Many of you may be asking yourself why I need fetchmail if I've already setup my own mail server (see the sendmail page). "Why not just set up a forward on your university server and send all e-mail automatically to your local one?" Well, there are a couple of reasons not to:

1. I like to have the freedom of having my own mail server, if I really need to use it in a pinch, but do not wish to use it unnecessarily to avoid potential trouble down the line. Mail that is sent from the local mail server via sendmail will be masqueraded as having come from @csufresno.edu to make it appear that everything is coming from a single location (to avoid the confusion of having e-mail sent from a single user come from multiple, although closely related, domain names) — even though it really would be coming from @optics.csufresno.edu. While this may not be an issue for most people, for some systems that have very tight spam controls, their system may be set up in such a way so as to reject incoming e-mail using masqueraded headers.

2. There are times that I need to bring my local server offline for a bit — whatever the reason may be (reboot the kernel, upgrade the system, etc). I would rather e-mail always be sent to the university server, where I know that it will arrive there safely and wait for me until I fetch it myself, than for mail to start bouncing all over the place if my local server is temporarily offline. But the nice thing is that with a local mail server, if the university server does go down, I can still send e-mail and tell people to send it to @optics.csufresno.edu instead so that it arrives directly to me. For all other mail retrieval, there is fetchmail.

Setup fetchmail and ~/.fetchmailrc

fetchmail is easiest to work with, in my opinion, if you use it in daemon mode. Setting up fetchmail in this way used to be as simple as adding a single line to /etc/rc.local to poll mail, but it appears that the current implementation of SELinux is having issues with this...

See www.linuxquestions.org/questions/showthread.php?t=444090 for details of the problem.

Fortunately, there is a very simple work around. Instead of setting up fetchmail to be executed in /etc/rc.local, we can set it up as a cron job that is executed once at boot time by editing /etc/crontab. Add the following line:

@reboot root su [user] -c "/usr/bin/fetchmail -d 60 -f /home/[user]/.fetchmailrc"

While booting, fetchmail will be launched as a daemon owned by [user], and will poll mail every 60 seconds (the "-d 60" option), as directed by the /home/[user]/.fetchmailrc file. This does, of course, mean that you will have to reboot the system to actually launch fetchmail (unless you do it the first time by hand). :( Here are the contents of the /home/[user]/.fetchmailrc file:

# Set fetchmail parameters
poll [server].[domain].[name] aka [alias].[domain].[name]
    protocol pop3
    user "[server user]" there is "[local user]" here
    password "[password]"
    ssl
    fetchall

mda "/usr/bin/formail -s /usr/bin/procmail -t -m /home/[user]/.procmailrc"

When you save and exit, make sure that only you have permissions to read and write the the file, since your e-mail password is present:

~> chmod go-rw ~/.fetchmailrc

Basically, the ~/.fetchmailrc file tells fetchmail to poll the remote server, which may also be known by an alias, using the POP3 protocol. My local user name here is different than the remote user name of the university's e-mail server, hence the "user ... there is ... here". Then I supply fetchmail with my mail server's password, and ensure that SSL encryption is used when polling. "fetchall" dumps all e-mail from the server to my local directory. The last line uses formail to forward the newly received e-mail through procmail (an autonomous mail processor) one e-mail message at a time (the "-s" option) so that it can be filtered through spambayes (my spam filter) correctly. I will discuss procmail more in detail below (in addition to when I discuss spam filtering on the spambayes page), but sufficient to say here, the "-t" option of procmail allows it to fail softly so that if the e-mail cannot be delivered, it will not bounce, but will return to the mail queue; the "-m" option uses my ~/.procmailrc file as the basis for it's processing and filtering rules. If you are not going to forward your mail through procmail, comment the last line out!

The exact syntax here for the ~/.fetchmailrc file is extremely important, as it is very easy to put yourself into multi-drop mode. This will cause all kinds of trouble and warning messages with the latest version of fetchmail (i.e., when using Fedora Core 6) if the remote mail server is not configured correctly. Multi-drop assumes that a single e-mail is supposed to be dropped off to multiple users, instead of a single user. If, instead of using the above syntax, you have something similar to:

user "[server user name]" is user "[local user]" here

fetchmail will assume you are multi-dropping. And since the remote University mail server at Fresno State is not configured correctly to hand multi-dropping, fetchmail will tack on the following warning in every single one of the headers to my incoming e-mail:

X-Fetchmail-Warning: recipient address [server user]@[domain].[name]
didn't match any local name

This is a problem, because it should! As a result, fetchmail just dumps the e-mail to whoever owns the process, which in this case, is me...so the e-mail still arrives...but far from correctly. Furthermore, the daemon will also complain and not run unless you override it using the 'envelope "Received"' option in .fetchmailrc. But even if the option is set, this still provides the X-Fetchmail-Warning warning message, because you are trying to multi-drop and the remote server does not know how to handle it. All of this because of the second "user" keyword that was used. For the many hours that I was trying to figure out what was exactly going on, life was not so good.

As you can probably tell, it took a long time for me to debug the problem. And I'm not the only one. Perform a Google search on the above fetchmail warning, and you will find a whole host of people who have started seeing the same thing, and a bunch of very irritated developers being asked about it all of the time. In general, this is all because either:

1. The remote e-mail server is not configured to allow for multi-dropping.

2. You have configured fetchmail incorrectly by telling it to multi-drop when it is not supposed to.

In my case (up until now), both were true. I cannot do anything about the University's remote e-mail server, so I finally tracked down the problem on my end myself. The proper syntax for setting up the system for single-drop works with any one of the following lines:

    user "[server user name]" is "[local user]" here
    user "[server user name]" there is "[local user]" here
    user "[server user name]" to "[local user]" here
    user "[server user name]" there to "[local user]" here

Any one of these four lines will properly configure fetchmail to be used in single-drop mode, and you will avoid the warnings and problems with the latest version of fetchmail not wanting to launch the daemon without the envelope option for multi-drop. No more warnings, and incoming e-mail is received correctly. Life is good.

Setup formail via ~/.forward

With spambayes fully installed, the next step is to setup a ~/.forward file in your home directory (/home/[user]/.forward) so that incoming e-mail is automatically forwarded through procmail by formail. This is done by adding the following line to the ~/.forward file:

"|exec /usr/bin/procmail -f-||exit 75 #[user]"

Save and exit.

Setup procmail and ~/.procmailrc

Unfortunately, the latest versions of sendmail do not allow mail to be forwarded through procmail by default. To correct for this, change into the /etc/smrsh directory and create a soft link to procmail:

~> cd /etc/smrsh
~> sudo ln -s /usr/bin/procmail .

Now restart sendmail:

~> sudo service sendmail restart

You should see the sm-client and sendmail clients successfully restart:

Shutting down sm-client:                                   [ OK ]
Shutting down sendmail:                                    [ OK ]
Starting sendmail:                                         [ OK ]
Starting sm-client:                                        [ OK ]

When procmail is launched, it checks for a local ~/.procmailrc configuration file located in your home directory. Create a /home/[user]/.procmailrc file with the following:

# Next may be needed if you invoke programs from your procmailrc
# Details in Check Your $SHELL and $PATH in Troubleshooting below
SHELL=/bin/sh

# Directory for storing procmail configuration and log files
# You can name this environment variable anything you like
# (for example PROCMAILDIR) or, if you prefer, don't set it
# (but then don't refer to it!)
PMDIR=$HOME

# Put ## before LOGFILE if you want no logging (not recommended)
LOGFILE=$HOME/.maillog

# To insert a blank line between each message's log entry,
# uncomment next two lines (this is helpful for debugging)
LOG="
"

# Set to yes when debugging
VERBOSE=no

# Remove ## when debugging; set to no if you want minimal logging
## LOGABSTRACT=all

# Replace $HOME/mail with your mailbox directory
# Mutt and elm use $HOME/Mail
# Pine uses $HOME/mail
# Netscape Messenger uses $HOME/nsmail
# Some NNTP clients, such as slrn & nn, use $HOME/News
# Mailboxes in maildir format are often put in $HOME/Maildir
# NOTE: Upon reading the next line, Procmail does a chdir to $MAILDIR
# and relative paths are relative to $MAILDIR

MAILDIR=$HOME/mail #Make sure this directory exists!
DEFAULT=/var/spool/mail/[user]

# Put everything else in the INBOX
:0:
$DEFAULT

Save and exit. For now, all mail that is routed through procmail will be dropped into the /var/spool/mail/[user] directory by default. We will later modify this file to pass incoming mail through the spam filter to classify each e-mail as ham, spam, or unsure. See the spambayes page for details.

In conclusion, "let me explain…no, there is too much. Let me sum up." fetchmail is used to fetch my e-mail, where it is then forwarded via formail one e-mail at a time through procmail so that it can eventually be filtered for spam by spambayes. There. That's not so complicated, is it?

References:

http://www.ii.com/internet/robots/procmail/qs/

I really need a couple of more here. But if you perform a Google search on fetchmail and procmail, you'll turn up about a million HOWTOs regarding syntax...

F11 E-Mail & Spam — fetchmail Configuration