Understanding Email

Posted on November 1, 2007 
Filed Under Mail

If you have ever set up an email program, you may have seen that it breaks the mail handling process into two parts — getting your mail from somewhere, and sending mail. The “getting” part is usually called POP (which stands for Post Office Protocol), while the “sending” part is called SMTP (Simple Mail Transfer Protocol).

Quite recently, Google has announce that their Google Mail service will also allow you to access your mail using IMAP — and this has a lot of folks excited.

We’re talking about replacing the POP half of the email setup process. Why? Because POP has some serious limitations. As a general rule, when you run a POP mailer, it has to move all of your mail from the mail server onto your local machine. That’s move, as in “make a copy and then delete the originals.” So if you want to have a copy on your desktop machine at home and also have a copy on the notebook that travels with you, you have a problem: Do you set one machine to get the mail and “leave mail on server”? Then fetch the mail on the other machine? What if you run them out of order?

No matter what clever workaround you come up with, it’s annoying, a hoop to jump through, and a risk that you’ll misplace mail that is important to you.

IMAP — Internet Message Access Protocol– doesn’t play the game that way. It runs on the server, which now becomes the machine where you want your mail to be archived forever (or near enough).

You then connect to the server with a client program, a mail reader, that “speaks” IMAP — and that is probably just a matter of changing one setting on your current program. But no matter how many different notebooks or desktops you use, as long as you only use IMAP, they will each see the same view of the mail on that server.

Just make sure you don’t have a notebook somewhere that still thinks it “owns” your mailbox and “POPs” all your mail off of the server. That spoils the plan.

Given a couple of gigabytes of free Google storage, you can see why accessing it this way makes sense.

Mail delivery

Another name for a mail server is an SMTP server. (”Psst! SMTP?” Simple Mail Transport Protocol.) A “server” is a computer (and a program) that listens for other machines and does something for them — accepts email from them, in this case, which it then writes to the disk that serves as your mailbox. The details vary, as you may imagine, considering that a big mail server might have thousands of users… or millions, if you are talking about AOL, Yahoo!, or Hotmail.

Hands On

Let’s stick with the more personal scale, shall we? A typical hobbyist or small-business box, running a form of Unix (I hope!) and an SMTP program such as postfix or maybe exim. If the machine’s Internet connection is reliable enough, you could make the decision that it will be your mail server, on the Internet, always on, always listening for incoming mail.

It could also be the machine that sends your outgoing mail, with a few precautions. If it’s on a local cable company, you probably would run into policy statements that home users are not supposed to run servers. The volume of mail you send is so small that no one should really care… but the number of hijacked “zombie” PCs that are spewing spam is so humongous that no one in his right mind is going to believe that your cable box is a legitimate server. Sorry; it ticked me off when people started rejecting my mail, too.

Assuming your server is on a commercial hookup somewhere, you still might have a problem getting mail to its SMTP port from home, for exactly the same reason as above — home cable user, trying to send mail to the SMTP port anywhere except the cable company’s mail server? Potential terrorist! Feh. So, you need to take things one step further — you need authenticated access to your SMTP port to keep out the spammers, and you need to hit it on a port other than the default, port 25. “Authenticated submission of email for relay” takes place on the submission port, 587. Properly set up, you should now have a mail server that you can rely on from darn near anywhere. Never again find yourself tearing out your hair in a strange hotel room because you can’t reach either your home cable company or your work server.

Having your own box is also a great thing for controlling spam — it can notice and act upon spam or viruses before they are delivered, simply by spotting nonsense (such as a remote machine that claims to be a member of your domain). This is also the only place that it is safe to do a “return to sender” rejection of spam or viruses.

Once mail has been accepted, it becomes the responsibility of the machine that has accepted it. If that machine can’t find another route to make the delivery, it bears the responsibility to bounce it back to the original sender… and since the original connection no longer exists, that means trusting the “From” or “Reply-to” address in the mail itself. Far too many system admins nowadays have not yet come to grasp that in spam, those addresses are fake.

Time for a rant. As a mail admin, your machine must decide whether it can deliver a piece of mail or not. before it allows the connection to close — and if it’s spam, and you can’t deliver it, then it needs to be Dropped. On. The. Floor. Not bounced to an innocent victim. Certainly not bounced back to ME, just because some lying sack of slime faked MY name in the From: address.

But we digress…

You have an SMTP server accepting mail that is being sent to you. (SMTP is a sending protocol, but a sender also implies a receiver.)

Thus, you will also want it to act as a POP or IMAP server, so that you also can log on to get the mail that has collected in your inbox.

Usually, the incoming mail goes directly to a special directory, for example /var/mail/yourname

Once you have read your inbox mail, it would then be filed away somewhere else, perhaps under the Mail subdirectory of your account’s home directory — or in Unix terms, /home/yourname/Mail/mbox

The most obvious difference between using POP and IMAP is that the server needs somewhere to keep your email, and to keep track of the various folders which your desktop email program uses. IMAP does this by creating subdirectories (or subfolders) under your Mail directory, some of which are visible to you and others which are hidden, and are used to keep track of which messages have or have not yet been read. This is how you can add (or “subscribe to”) a new subfolder on your desktop, and have it show up in another IMAP client on another computer the next time you log on there.

Now, this assumes that you have a mail server of your own “on the Internet.” That may not be case, but you can still get many of the benefits of having such a setup, even if your Internet provider is someone you call up twice a day over a telephone line connection. How would that work?

  1. You run a package called fetchmail on your local Linux box.
  2. It logs on to the remote mail server, picks up your mail (with POP or IMAP, either one), and hands it over to the local mail program, such as postfix or exim. The only real difference is that fetchmail takes the place of the always-on, always-listening SMTP connection. The postfix program and IMAP server programs are identical.
  3. Finally your local IMAP daemon makes your mailboxes available…
  4. … to your mail reading program of choice.

Note that you can apply other tools such as procmail, spamassassin, and Spam Bouncer to the mail as it is handed off from fetchmail to postfix.

In addition to the postfix (SMTP) program, you still need a program to make your mail available to your mail reading program — which is where a local IMAP program comes into it. I rather like Dovecot myself. It provides both POP and IMAP services, and you can use just about any Internet mail program with it. Eudora, Thunderbird, TheBat, Pegasus, Outlook — that doesn’t matter. You just tell it where to find your imap server and how to log in to your account.

Dovecot offers different strategies for how your mailbox might be set up; I’m going to stick with this one, simply because it’s the one I described above:

# default_mail_env = mbox:~/Mail/:INBOX=/var/mail/%u

Or in other words,

So, we uncomment this config line, as found in /etc/dovecot/dovecot.conf, by taking off the leading hash mark:

default_mail_env = mbox:~/Mail/:INBOX=/var/mail/%u

These lines in my .procmailrc make it agree with Dovecot about how things will proceed:
MAILDIR=/var/mail #you’d better make sure it exists
DEFAULT=$MAILDIR/username

After this, all my “delivery” recipes for procmail could end with “$DEFAULT”, but one of the big advantages of procmail is that you don’t have to fish everything out of your INBOX. You can also have mail delivered directly into a folder without ever stopping in the Inbox — you can have $HOME/Mail/important or $HOME/Mail/spam.

Postfix and Procmail

As hinted above, procmail can change where mail ends up, and in fact you can tell Postfix to hand all mail off to procmail for the actual delivery — it’s nice to set that up system-wide if you have more than one user, because those who set up a .procmailrc file can set up filters as they please, while those who do not set up .procmailrc files won’t really notice any difference; the default behavior of procmail will be to simply deliver the mail as usual.

If you want to enable procmail system-wide, just uncomment the appropriate line in your /etc/postfix/main.cf file, then do a postfix reload command as root.

If you have the procmail binary installed but don’t have Postfix (or exim, or sendmail) configured to use it, you can still take advantage of it by installing a .forward file. For more about procmail, see Procmail 101.

Mutt

All of those graphical email reading programs have their place, but there’s a lot to be said for the simplicity of the console, and the premier console-based email program, Mutt.
If you are willing to read your mail using Mutt exclusively, its defaults should work just fine for you. You may have noticed that Mutt asks you a question upon exit — it wants to know about saving your mail from your default spool file (which we call /var/mail/yourname) by copying it to a mailbox called “mbox” (which is /home/yourname/Mail/mbox). You can turn off the prompt (”Move read messages to mbox?”) by changing a flag in your .muttrc — it’s set to “prompt” or “ask” by default, but you can change it to yes or no. Mutt has a lot of these three-valued “yes/no/ask” flags; a bit of poking around will quickly make it clear how they work.

If you’d like to drag-and-drop mail between folders, you have options for a Graphical User Interface (GUI) IMAP-aware mail client. Apple’s Mail.app (the .app is hidden, but that’s its real name), Thunderbird (from the Firefox people), KMail under KDE. When you set up your client it will talk to your imapd (”imap daemon” is the the role being played by Dovecot) to find out where your folders are, and will offer to let you “Subscribe” to specific ones. You should not have anything in your Mail directory except mailboxes, to avoid confusion, but you can get away with having a directory with mailboxes in it; a good mail program will be able to cope with that (and present a useful tree structure of folders within folders).

When you log on for the first time, your IMAP-aware mail reading program should show you an Inbox. If you’re starting from scratch, you will have an empty Inbox which is physically located in /var/mail/yourname, and an empty Mail sub-directory inside your $HOME directory. Just for practice, you can tell your mail-reading program to create a new folder inside /home/yourname/Mail, such as “Important”. If you’re using mbox format, this will create a flat file, /home/yourname/Mail/Important.

When setting up the imap daemon (server program), you may be asked where the “base” directory is — in our example, that is /home/%u/Mail (the %u stands for “username”). It’s also common to use /home/%u/mail with a lower-case “m”. Either way, though, it’s best if you don’t fight with the system about where INBOX is. That’s best left in the system realm, which means over in /var/spool/mail or /var/mail.

Often /var/mail and /var/spool/mail will actually be the same place, by the way — the system will set up both, just to placate any programs that expect to find one or the other, and will do it in such a way that it doesn’t matter which name you use.

Once you have a desktop mail client talking to an IMAP server, you also have some other options. Since your mail stays on the server, you can run your mail program on a notebook, and still reach your saved mail securely from darn near anywhere. That’s a forthcoming article, about setting up a “poor man’s VPN” to get to your home machine securely as though you were right there.

Hint: Google for “SSH tunnels”!

Comments

Leave a Reply

You must be logged in to post a comment.