As mentioned in the previous post in the Postfix series we now turn our attention to how our Postfix setup is perceived by other MTAs.
Incoming and outgoing mail have tended so far to generate separate topics and posts in this series. But reducing spam and not getting taken for a spammer are to an extent two sides of the same coin. The easiest to use tools for both are the PTR and SPF records. To reduce spam, you check them for incoming connections. To prove – or make it more plausible – that you’re not a spammer, you set them.
PTR records
I was first put on to the topic by another guide that had a spam reduction tip that made me wince: PTR records. If the calling SMTP server doesn’t have PTR records, it suggested, you should reject the incoming mail or in fact, the connection outright. Did I have PTR records for my server? After some quick searching and testing, it was clear that I did not.
So what is a PTR record? The easiest way to explain it is with an example of an incoming connection:
Oct 30 21:39:17 aaac736a83e6 postfix/smtpd[120]: connect from unknown[114.237.109.144]
Oct 30 21:39:17 aaac736a83e6 postfix/smtpd[120]: DC5FF2C15F7: client=unknown[114.237.109.144]“Connect from unknown” means no PTR record was found. Postfix knows the IP address of the calling server (or client in this instance) but it is unable to do a reverse DNS lookup in order to identify it. I want to know who is calling and they should be able to identify themselves properly. Sure, it can go by the client HELO introduction but that more about formality than security. A server can claim to be pretty much anyone there.
Do reverse DNS/PTR lookups work?
A note of caution here: If the reverse DNS lookup fails for any reason you will get “connect from unknown” for any and all clients. “Any reason” includes those cases where the Postfix smtp daemon is unable to use the right libraries, which it often will be if it’s running chrooted (i.e. without access to the wider file system). And at least on my current distro, Debian 10, Postfix seems to come with chrooting on by default for a number of services, including the smtp daemon. Dave Goodwin has more on that. For me the solution was to copy the whole library directory /lib/x86_64-linux-gnu/ wholesale into /var/spool/postfix/lib/ along with some pertinent conf files (resolv.conf and nsswitch.conf from /etc into /var/spool/postfix/etc). Possibly not ideal but probably better than disabling chroot. An easy way to test if lookups fail when they shouldn’t is to compare the result from Postfix logs to that of MXToolBox. If Postfix says “unknown” when MXToolBox’s PTR search finds a domain name to match the IP address, Postfix is probably missing some tools.
Postfix relay restrictions
Once my logs were consistently reporting the correct hostnames whenever contacted by smtp clients, I added one more to my list of smtpd_relay_restrictions:
smtpd_relay_restrictions = permit_sasl_authenticated
reject_unknown_reverse_client_hostname
defer_unauth_destinationreject_unknown_reverse_client_hostname is a ‘soft’ requirement: All it demands is that the client’s IP address returns a host name, i.e. that PTR records exist. A quick rundown of the spam mail that I receive a surprisingly large number of mails fail this simple test. Of course, a large number of homebrew mail servers probably also fail it – I know mine did for a long time – so I might be punishing my own kind as well. Time will tell. Here’s an example from the logs of the rule in action:
: connect from unknown[37.49.230.117]
Aug 13 10:22:06 brokkr postfix/smtpd[336]: NOQUEUE: reject: RCPT from unknown[37.49.230.117]: 450 4.7.1 Client host rejected: cannot find your reverse hostname, [37.49.230.117]; from=<spameri@tiscali.it> to=<spameri@tiscali.it> proto=ESMTP helo=<win2012r2RDP>Double checking with MXtoolbox and yes, 37.49.230.117 does not have any PTR records, so Postfix is correct. A connection from a spammer who has set up PTR records proves that the name lookup is working
Aug 13 07:46:59 brokkr postfix/smtpd[315]: connect from hwsrv-752331.hostwindsdns.com[142.11.205.237]Because hwsrv752331 tries to use Postfix as an open relay it gets rejected anyway but not because of PTR records. Why don’t all spammers have PTR records? I don’t know. Maybe because domains cost money and are easy to put on ban lists? But then I can put any domain in my PTR record, gmail.com for that matter. It’s only when I also do the reverse reverse DNS lookup, i.e. a DNS lookup, that liars get caught out. Which leads me to…
A more stringent requirement than just the presence of PTR records is that the resulting domain name from the PTR lookup should itself have a DNS A record – and that the IP adress from that record should match the original IP address. Postfix will warn about this when using the above restriction but using reject_unknown_client_hostname (without the reverse) it will turn it into a requirement.
What about my own PTR records?
On the related topic of ‘how to ensure that I myself have set my own PTR records?’, I went to check first with my domain registrar who redirected me to my VPS supplier. Figuring out which record goes where can be tricky but I can see some logic here. The DNS record authoritatively says where my domain is hosted so it belongs with my domain registrar. The PTR record speaks to where my IP address belongs so it belongs with my VPS host who is lending me the IP address. The two mutually support and complement each other. Sort of like looking up a phone number in the directory and then calling said number and asking if this is truly [insert name that I looked up].
The MX record, for the record, is just a specification of what (sub)domain the mail server lives on, since most people don’t use one server for everything like me.
At my VPS host I found a “Reverse DNS” link in the main menu. Easy as can be. A lot of guides suggest setting up DNS zones prior to doing PTR records but I have not found this to be necessary (AFAICT DNS zones are a bit of an advanced topic). After a couple of hours the new PTR records were reported by tools such as nslookup, dig and host.
Quick recap: DNS, MX, PTR and SPF
If you’re anything like me, one of the things that has had you considering giving up on selfhosted email is the sheer jungle of these records and the difficulty of keeping them straight in your head. So let me see if I can make this simple.
If PTR records are the inverse of DNS records, SPF records are sort of the inverse of MX records. Remember those? MX records tells an SMTP client which server is the allowed recipient of an email for a specific domain. An SPF record, on the other hand, tells an SMTP server which clients are allowed to send email on behalf of a domain. Where DNS/PTR is about identity (are you who you say you are?), MX/SPF is about permissions (are you allowed to do what you’re trying to do?)
When it all boils down to one domain name and one IP, this can seem overengineered. But if thousands of servers are allowed to send and receive email for gmail users, identity and permissions are clearly very different things.
SPF records
Setting my MX record was simply entering in my main domain name. Lookup the DNS A record for “brokkr.net” and you will find the host allowed to receive email on behalf of the domain “brokkr.net”, it said. SPF records are a bit more finicky and have more options but I’m not going to cover them all. Instead I will cover this one example which should do for 99% of all selfhosted setups.
v=spf1 mx -allApart from the initial version specification (v=spf1) every element in the line, divided by a space, signifies a host or group of hosts AND what permissions it/they have. In this line, there are two elements, “mx” and “-all”. If we are an MTA with an incoming connection purporting to deliver email from a specific sender (“alice”) from a specific domain (“brokkr.net”), we look up brokkr.net’s policy and read it left to right and try to determine if we think the sender is legit or not.
“mx” is short for “+mx”, the plus is implied if there are no other preceding signs. The plus sign is the permission, same as the minus sign in the second element. “mx” is the domain name or group of domain names that the sender has listed in their MX records. Plus means that the host or group of hosts are allowed to send email on behalf of the domain. Minus means that they’re not.
Back to the scenario where we are seeing an incoming email. The policy for brokkr.net says that if the sender is one of the hosts mentioned in the MX records, they are allowed to send. Simply put: If the host is a legitimate destination for email for brokkr.net, it is also a legit sender of email for that domain. Obviously we now have to lookup the MX records for brokkr.net. Since MX just says “brokkr.net”, we now lookup the DNS A record for “brokkr.net” and check that against the incoming connection. I could also have skipped the MX part and substituted “mx” for “a” as the group, i.e. all hosts that match the domain name’s A record. If it feels like retreading ground from the PTR section, remember that this is for giant corporations where these jobs are spread over many, many hosts. If there is a match, the sender is allowed to deliver email.
Suppose instead the sender fails that check. It is not included in the MX records. Then we move on to the next element. Which could have been another set of allowed domains or IP addresses. Instead it’s “all”, i.e. a match against anything and everything. And, unsurprisingly, this group is not permitted to send email on behalf of the domain.
Setting policy
To keep things straight: I want to set the policy for my own domain so that other senders may not impersonate me. Checking other senders is about making sure they’re not impersonating someone else, i.e. fraud or more likely spam. First, I will set my own policy.
Like DNS and MX records, SPF is considered part of the domain’s policy. We see the sender’s purported domain and so we go check with the registrar of that domain. So in order to edit my own SPF records, I check in with my registrar and add it. With Namecheap, my current registrar, SPF records are considered a subset of generic TXT records. So I add one of those and copy paste the text above into the line and make sure that it applies to the naked TLD domain (“brokkr.net”). Obviously this will vary from registrar to registrar. If any do not offer it, I would suggest changing registrar. As always with anything records related, it will likely need time to propagate.
Because I had badly configured my MX records – and the SPF records relied on the MX records – I got a chance to see how other MTAs react to this. Badly, is the answer. My email got classified as spam, pure and simple:
Received-SPF: fail (oxse3.privateemail.com: domain of brokkr.net does not designate 167.86.120.123 as permitted sender) client-ip=167.86.120.123; envelope-from=mail@brokkr.net; helo=brokkr.net;
X-SPF-Result: oxse3.privateemail.com: domain of brokkr.net does not designate 167.86.120.123 as permitted sender
...
X-SpamExperts-Class: phish
X-SpamExperts-Evidence: SPF
X-Recommended-Action: rejectOnce I had fixed it, things looked well, better, definitely:
Received-SPF: pass (oxse2.privateemail.com: domain of brokkr.net designates 167.86.120.123 as permitted sender) client-ip=167.86.120.123; envelope-from=mail@brokkr.net; helo=brokkr.net;
X-SPF-Result: oxse2.privateemail.com: domain of brokkr.net designates 167.86.120.123 as permitted sender
...
X-SpamExperts-Class: unsure
X-SpamExperts-Evidence: Combined (0.37)
X-Recommended-Action: acceptEnforcing policy
Enforcing SPF is a bit more complex than setting the policy and also more than checking PTR records because there isn’t an inbuilt SPF check policy in Postfix. For that we have to turn to plugins. Fortunately (at least for containerization purposes) it is not a daemon but a process that can be controlled and run by Postfix itself.
After installing postfix-policyd-spf-python (package name on Debian based systems), I add a process to Postfix’s master.cf list:
policyd-spf unix - n n - 0 spawn
user=policyd-spf argv=/usr/bin/policyd-spfSince the column headers are cut out, this can be difficult to understand so I will try to explain the pertinent bits. The spawn on the far right is the command type. Spawn is a daemon running under Postfix, that will, you guessed it, spawn other processes, in particular those external to Postfix itself. Spawn is handed both the user to run that process as and the binary path as variables, user and argv, respectively. The unix bit indicates that the process spawned can be reached via a unix socket.
Fortunately, even if Postfix does not have an inbuilt SPF checker, it does know policy checkers as a genre and what to expect from them. So now, all it takes to use my new process is to add yet another relay restriction:
check_policy_service unix:private/policyd-spfNote that most guides advocates adding this as a recipient restriction, rather than a relay restriction for fear that it may interfere with sending out email. But I only send out email from my submission process, defined in the last post, and that comes with it’s own overriding smtpd_relay_restrictions list. It might restrict the system’s own ability to send out email, e.g. MAILER-DAEMON type notices, but for now, I will live with that. Note that the policy argument simply tell check_policy_service where to find the socket for the policy software, not what it is or what to expect from it. Presumably then, the result is fairly simplistic, like other relay_restrictions: To reject, defer or accept.
Adding the SPF check went surprisingly smoothly for me (read: I didn’t have to make 10+ changes to configuration before it worked). I have yet to see it actually reject anything but its presence can be seen in an added header to received mail:
Received-SPF: Pass (mailfrom) ...Finally
I think these two checks should do away with most of the spam that I recieve – and put me in sufficiently good standing to be allowed to email most people on the internet. But we can always be more salonfähig so hopefully I will get around to DKIM at some point, though I am probably putting it off till last.
Meanwhile, it’s finally time to head back to Dovecot. When we left our feathered friend, it was still very much a fledgling. We are now going to a) have it take over some mail handling duties from Postfix (LMTP) and b) equip it with an advanced filtering engine (Sieve). Since the latter depends on the former, it makes sense to do them together. After that we will return to Postfix to deal with the internet’s preferred processed pork product, spam.
Fingerprint by Alex Borland (edited by brokkr.net), CC0 Public domain
