MUAMSAMTAMDAMAAAAAA! Understanding how email works so you can get on with setting up your own mail server

‘Email infrastructure’ is a word most people never want to hear. And with good reason. The elements and procedures involved in getting email from user A to user B are legion and complex. And that’s just the theory before you even start to get into the practical implementation side of things. However, understanding the theory will make a more practical how-to (like this one for Arch Linux) easier to follow.

This article will offer an introduction to that theory. There are plenty of ressources to draw on, so mostly this is simply a meta article with links to the best of the internet on the various sub-topics with a bit of filler where needed. The aim is not give an in-depth understanding of how, say, SMTP works. The aim is to provide the wanna-be master of his own email domain the minimum of theoretical grounding to accomplish that goal.

Note: This introduction assumes you know what terms like server, client, protocol, and ports mean. At least roughly. Ideally you have some experience setting up other services for network use, such as a web server.

Agents

Agents are the ones carrying your mail back and forth, the various specialised postmen and drop offs of the internet to use an analogy that will become hauntingly familiar to you once you’re done reading this. I recommend first going to the Wikipedia’s article on email agents. It’s really just an advanced disambiguation page that links to the articles on various kinds of agents but it quickly gives you a framework for understanding:

  1. That a lot of elements are involved in getting mail across the internet. The unified notion of a ‘mail server’ is either a figure of speech or a dedicated piece of hardware, not a piece of software that you can click-and-install.
  2. That each function does not necessarily correspond to one distinct piece of software. Sometimes software packages only do part of a distinct function and sometimes they incorporate more than one function.

Make a quick note of the MxA acronyms and keep the page open for when you feel the need for more detailed info on any one agent in particular. But for better understanding I recommend first going to this article on Kioskea.net for a simplified introduction to the main components of the setup: User agent (MUA), transfer agent (MTA) and delivery agent (MDA). The best part of it is this:

To use a real-world analogy, MTAs act as the post office (the sorting area and mail carrier, which handle message transportation), while MDAs act as mailboxes, which store messages (as much as their volume will allow) until the recipients check the box. This means that it is not necessary for recipients to be connected in order for them to be sent email.

The article leaves out two acronyms, probably because these functions tend to be subsumed into others: The submission agent (MSA) and the retrieval agent (MRA). The former accepts outgoing emails from the MUA and hands them onto the MTA but is usually part of software classified as MTAs (e.g. postfix). The latter is in all but classic Unix CLI interfaces part of an MUA: The part of the client that retrieves email from the MDA using either POP3 or IMAP. To get a better understanding of these as well as the others, I recommend the aptly named MailConcept page from the Mutt project (an MUA or email client).

Personally I had the most difficulty understanding the role of the mail delivery agent, the MDA. What exactly did it do? If we keep the post analogy, the MTA stuffs the mail in the letter box (the MDA) and there the user (the MUA/MRA) comes and gets it. Being a letter box doesn’t sound like a lot of work. Why is this a function (and invariably a program) in itself? Couldn’t you simply get your mail at the post office, i.e. the MTA, so to speak? This is probably partly a Unix philosophy thing: Do one thing and do it well. And in large corporations there are probably advantages to having separate machines deal with outside and inside requests. But apart from that there are a few hidden tasks that the MDA takes care of: Local storage in an appropriate format, filtering, sorting, spam and virus checks, subtle fixes to encoding etc. And depending on the protocol used (POP3/IMAP) the MDA also tracks state changes such as read/unread or deleted. In other words, if the MDA is a letter box it’s a pretty clever one.

Protocols

Okay, so now we know all the individual participants involved, the agents. But what about how they talk to one another. To gain a basic understanding of how MTAs talk to one another this article gives a decent and very beginner friendly introduction to SMTP. Helpfully, it also maintains the same analogy of post offices as the Kioskea article and importantly, the analogy stays on the same level, i.e. post office means MTA (which means that SMTP must be the mail train).

For anybody used to configuring an email client for gmail or the like, SMTP tends to get associated with the connection between your client and the server for outgoing mail. While this connection also uses SMTP as a protocol, it’s important to keep in mind that in the context of setting up your server, SMTP is mainly about servers talking to servers about mail coming and going, not about clients (MUAs) and not only sending. This is the distinction between submission and relay that was introduced in the late 90s.

IMAP and POP3 on the other hand are protocols used between the MUA and the MDA (or more precisely, the MRA and the MDA). It is virtually (not literally) impossible to find a decent article about IMAP that doesn’t contrast it with POP3 from a consumer choice perspective (and vice versa) but the Email protocols section of the Wikipedia IMAP article comes close. And as for making that choice this article really has all you need to know that POP3 does not make much sense in this day and age. Unless you don’t trust your own server to hold your mail. Which there might be cases for but if that’s you, then you should probably go stuff your savings in a mattress as well.

DNS and MX records

If the SMTP protocol are the mail trains, DNS and MX records are the railway map. Or maybe the train table. This analogy is getting tired. If you have set up a web server before this you probably already have a DNS record pointing your domain your IP address. [For the record, no pun intended, a DNS record is a linking of your domain name, e.g. brokkr.net, to your ip address] However, you may or may not know that DNS records are only part of how mail is directed to it’s proper destination (at least that’s not the correct way of doing things). MX records are consulted by the MTA when relaying messages and contain a prioritized list of MTA addressess for a given domain. As an example we send off an email to foo@bar.org. The MX records for bar.org are consulted and it turns out that there is only one MTA address listed for bar.org, namely mail.bar.org. Only then does the MTA look up the DNS record for mail.bar.org. There is a more detailed explanation on wireload.net.

Ports and encryption

When I was starting to understand basic web server functions I used to picture ports as these thousands of tiny bureaucrats, each sitting at his own numbered desk, resolutely refusing to deal with anything that wasn’t specifically his responsibility and not offering any kind of help or guidance as to the proper place for your inquiry. It works, right?

Everything you need to know and understand about the proper ports (including encrypted and non-encrypted) for the various protocols is right here, including a short history of outmoded practices that you might have come across on your way. It’s final words on port 465 versus port 587 seems to date it somewhat, though. For a more updated take on that read this article on Mailgun (but ignore the consluding pitch for a proprietary alternative to SMTP mail submission, “To SMTP or not to SMTP”).

That’s it. I know there are more direct routes to getting your own email setup but getting the theory straight first will help you immensely understanding what role the various configurations and installations play in the scheme of things.

Photo by Smithsonian Institution

Leave a Reply

Your email address will not be published. Required fields are marked *