So You Decided to Host Your Own Email Server (Or Update One)

A friend of mine decided to leave Gmail last summer. Privacy reasons, mostly — a few "Google read my drafts and showed me ads about it" moments that finally tipped him over the edge. He bought a cheap VPS, installed Postfix and Dovecot from a tutorial, pointed his domain at it, and within two weeks he was emailing me to ask why his messages to Outlook addresses were silently disappearing into the void.

Self-hosting email in 2026 is one of those decisions that looks reasonable at 11 PM and terrifying at 9 AM. The protocols are old, the documentation is older, and the vendors who own most of the inboxes on the internet (Google, Microsoft, Yahoo) decide whether your mail gets delivered based on rules nobody publishes. It's also entirely doable, and there are very good reasons to do it — you just need to know what you're walking into before you type

apt install postfix

This is post 1 of an 8-part series on email security. We're going to read the C source of Postfix and Dovecot, capture SMTP traffic at the byte level, walk through twenty-five years of CVEs by pattern, and finish with a checklist for pen-testing your own mail server. Before any of that, though: what does it actually mean to run your own email?

Why this series exists

Attackers already know how Postfix parses commands. They know which Dovecot versions have pre-auth bugs. They know that "I deployed it from a tutorial" usually means three misconfigurations and a missing DMARC record. The asymmetry between attacker and defender knowledge is the entire ballgame in security, and it's almost always the attacker who has more.

The argument for posts like this isn't "we're teaching attackers." Attackers learned this from RFCs, from the source code, from books published twenty years ago, and from each other. The argument is that between an uninformed defender and an informed defender, the informed one always wins. This series exists for the defenders.

The four daemons (and why everyone confuses them)

When you "run an email server," you're actually running three or four cooperating programs:

• MTA (Mail Transfer Agent) — accepts mail from other servers, decides whether to keep or relay it, and sends mail to other servers. Postfix is the most common open-source MTA. Sendmail, Exim, and OpenSMTPD are alternatives.
• MDA (Mail Delivery Agent) — takes mail the MTA has accepted and writes it into the user's mailbox on disk. Often built into the MTA, but Dovecot's LMTP and LDA components are commonly used here too.
• IMAP/POP3 server — serves the mailbox to mail clients (Thunderbird, iOS Mail, Outlook). Dovecot is the dominant open-source choice in 2026.
• MUA (Mail User Agent) — the email client itself. Not your problem as a server operator, but it's the thing speaking to your IMAP and SMTP daemons.

Most modern setups run Postfix as the MTA, Dovecot as the IMAP/POP3 server, and Dovecot's LMTP doing the MDA part. That's what the rest of this series will assume.

The protocol stack at thirty thousand feet

Mail travels across the internet on three protocols (well, two and a half), and your server speaks all of them on different ports:

| Port | Protocol | What it does | |------|----------|--------------| | 25 | SMTP | Server-to-server mail transfer (the relay layer) | | 587 | SMTP submission | Authenticated mail submission from a client, with STARTTLS | | 465 | SMTPS | Authenticated submission with implicit TLS | | 110 | POP3 | Mail retrieval, typically download-and-delete | | 995 | POP3S | POP3 with implicit TLS | | 143 | IMAP | Mail retrieval with server-side folder state | | 993 | IMAPS | IMAP with implicit TLS |

Post 4 in this series walks through SMTP, IMAP, and POP3 byte-by-byte. For now, just know that "an email server" really means "four or five daemons, each speaking one or two of these protocols on specific ports."

Why your ISP blocks port 25

Try to send mail from a residential IP and you'll discover that your ISP has quietly closed outbound port 25. This isn't malicious — it's the lesson of twenty years of spam. Botnets running on compromised home computers were responsible for a huge fraction of global spam by the mid-2000s. ISPs blocking outbound 25 was the most effective single intervention.

The consequence: you cannot self-host email on your home connection. You need a VPS or a colocated server with a dedicated IP, and even then you need to confirm with the provider that port 25 is open in both directions. Hetzner is famously strict — you have to open a support ticket and explain why you need it. AWS, GCP, and Azure block outbound 25 by default and require you to use their managed mail relay or move to a VPS provider that doesn't.

Even with port 25 open, your IP has a reputation problem. New IPs assigned to VPS providers often live on Spamhaus's PBL (Policy Block List) by default, because that's where botnets live. You'll need to delist your IP and warm it up gradually before Gmail and Outlook will accept your mail.

DNS records you cannot skip

The single biggest cause of "my self-hosted mail doesn't get delivered" is missing or misconfigured DNS. You need all of these, no exceptions:

• MX record — tells the world where to send mail for your domain. Points to your server's hostname.
• A record — your server's hostname must resolve to your IP.
• PTR record (reverse DNS) — your IP must resolve back to the same hostname. Major receivers will reject mail from a server with a missing or generic PTR (think
  Copy

  ec2-1-2-3-4.compute.amazonaws.com
  ). You configure this with your VPS provider, not in your DNS zone.
• SPF (Sender Policy Framework) — a TXT record listing which IPs are allowed to send mail for your domain. Without it, Gmail's spam filter gets very unfriendly.
• DKIM (DomainKeys Identified Mail) — your mail server signs outgoing messages with a private key; the public key is published as a TXT record. Receivers verify the signature.
• DMARC — a policy record telling receivers what to do when SPF or DKIM fail. Start with
  Copy

  p=none
  (monitor only) and tighten over time.

Skip any of these and major providers will silently filter your mail to the spam folder, or reject it outright. There's no "here's why we rejected your message" feedback — it just disappears.

Gotcha: SPF has a 10-DNS-lookup limit per record. If you've ever included Google, Microsoft, and three transactional senders in your SPF, you've blown past it without realising. The override is to flatten the record (resolve all the includes once and inline the result) but then you have to maintain the flattened version when those upstream providers change their IPs.

A minimal stack

Here's what a functional self-hosted setup looks like, in order:

1. VPS with port 25 open, a dedicated IP, and a sane PTR record
2. Domain with MX, A, SPF, DKIM, and DMARC records published
3. Postfix configured as MTA, listening on 25 and 587
4. Dovecot configured as IMAP/POP3 server on 993/995, plus LMTP for local delivery
5. OpenDKIM signing outgoing mail
6. TLS certificates from Let's Encrypt for both Postfix and Dovecot
7. Fail2ban (or equivalent) watching the mail logs for brute-force attempts
8. A spam filter — usually rspamd, occasionally SpamAssassin

A working install on a fresh Debian box is roughly:

apt install postfix dovecot-imapd dovecot-pop3d \
            dovecot-lmtpd opendkim opendkim-tools rspamd \
            certbot fail2ban
# (add postfix-mysql only if you're using a MySQL backend for virtual users)

This series isn't a step-by-step install tutorial — there are dozens of those, of varying quality. Mail-in-a-Box, iRedMail, and Mailcow bundle the whole stack into managed installers if you want a shortcut. What we'll do instead is read the source of what you just installed, understand its attack surface, and learn how to break and fix your own deployment.

The reality check

Before you start, three uncomfortable truths.

Deliverability is harder than configuration. You can have a perfectly secure, perfectly configured mail server that delivers nothing because your IP is on a blocklist or your domain is too new. Gmail and Outlook control most consumer and business inboxes between them, and they decide. Plan for weeks of warming up, monitoring blocklists (

mxtoolbox.com

mail-tester.com

Abuse handling is a job. If a user account on your server gets compromised and starts sending spam, you have hours to notice before your IP is listed everywhere. Self-hosting means you're now in the abuse-handling business — which means alerting on outbound mail volume spikes, monitoring

postqueue -p

Backups matter more than you think. Mail accumulates. People expect their messages from 2009 to still be there. Maildir is easy to back up because each message is its own file; mbox locking can make backup snapshots painful. Post 5 covers the storage layer in detail.

If you're still in, good. The next seven posts will leave you with more knowledge about how Postfix and Dovecot actually work than 99% of the people running them.

What's coming

• Post 2 — Postfix architecture: reading the C source to understand the multi-process design and what each daemon's attack surface looks like.
• Post 3 — Dovecot architecture: the same treatment, with a focus on the privilege-drop on login and the auth subsystem.
• Post 4 — SMTP, IMAP, and POP3 at the byte level, with Wireshark captures.
• Post 5 — How mail actually sits on disk: mbox vs Maildir vs mdbox vs database backends, plus encryption at rest.
• Post 6 — Twenty-five years of email server CVEs, organised by bug class so the patterns become visible.
• Post 7 — The misconfigurations that bite operators in 2026, with the specific config lines that fix them.
• Post 8 — A 30-item checklist for pen-testing your own mail server, focused strictly on the mail-layer attack surface.

You don't need to read them strictly in order, but posts 2 and 3 are the cornerstone — most of what comes later assumes you've read the architecture deep dives. Either way, by the end of this series you'll know more about what's listening on port 25 than the person who installed it.

What I'd Tell My Past Self

If I had to pick one thing self-hosters get wrong, it isn't the install — it's that they treat email like a static system. You install it once, it works, you stop thinking about it. Six months later, your DMARC report shows a third party spoofing your domain, your TLS cert renewed but Postfix didn't reload it, and a CVE landed for Dovecot two weeks ago that you didn't notice.

Mail servers are not appliances. They're long-running services that need attention. If you can't commit to checking your DMARC reports monthly, monitoring your blocklist status weekly, and patching within a few days of a security release, you don't want to self-host email — you want a managed provider with someone else's name on the abuse contact. There's no shame in that.

But if you're going to do it, do it knowing what's underneath. Post 2 is where we open up Postfix.