Pen-Test Your Own Mail Server: A Mail-Layer Hardening Checklist

This is the post the rest of the series was building toward. Posts 1 through 7 explained how mail servers work, what their attack surface looks like, and where misconfigurations happen. This post turns it around: now you attack your own server, with the goal of finding what you missed before someone else does.

A few ground rules before we start.

Your own server, only. Every command in this post is something a security firm would happily run during a paid engagement, and something a random person on the internet running it against your server would be a federal crime in most countries. Run these against:

• A server you own
• A server you have written authorization to test
• A test VM you spun up specifically for this exercise

Not your friend's server. Not your employer's server without written approval from someone authorized to give it. Get authorization in writing. If you can't get authorization, you can't test.

Mail-layer focus. This checklist deliberately ignores SSH, the host OS, web admin panels, and general server hardening. There are good guides for those elsewhere. We're testing the mail-layer attack surface — what an attacker reaching only ports 25, 110, 143, 465, 587, 993, and 995 can do.

Tools you'll need. Install on a separate test machine, not on the target:

sudo apt install nmap swaks netcat-openbsd dnsutils \
                 ncat openssl mailutils opendkim-tools
# testssl.sh from github.com/drwetter/testssl.sh
# (no apt package, clone it)

The 30-item checklist is at the end. The body of the post is each test in detail.

1. Recon: nmap

Start by seeing what the world sees:

nmap -sV -p 25,110,143,465,587,993,995 mail.example.com

What to look for:

• All expected ports open, none unexpected
• Service banners that don't leak version numbers (
  Copy

  Postfix 3.6.4
  is more help to an attacker than
  Copy

  Postfix
  )
• No unexpected services on adjacent ports

Gotcha: Postfix's banner is controlled by

smtpd_banner

$myhostname ESMTP $mail_name

smtpd_banner = $myhostname ESMTP

…to drop the software name. This is security-by-obscurity — it's not real defense — but it does cost attackers a small amount of time to fingerprint.

2. TLS audit: testssl.sh

./testssl.sh --ip mail.example.com:25 --starttls smtp
./testssl.sh mail.example.com:993
./testssl.sh mail.example.com:465
./testssl.sh mail.example.com:587 --starttls smtp

What to look for:

• No SSLv2, SSLv3, TLSv1.0, TLSv1.1
• No weak ciphers (RC4, 3DES, EXPORT, NULL)
• Certificate is valid, not expired, matches the hostname
• DH parameters are at least 2048 bits
• HSTS where applicable (less common for mail, but worth checking on the webmail front-end if you have one)

The output is detailed; read it carefully. testssl.sh flags issues by severity color — anything red is something to fix today.

3. Open-relay test

The seven canonical open-relay tests, from a separate external IP (not localhost, not from

mynetworks

# Test 1: Direct relay attempt
swaks --to victim@elsewhere.com --from attacker@external.test \
      --server mail.example.com:25

# Test 2: Spoofed local sender
swaks --to victim@elsewhere.com --from somebody@example.com \
      --server mail.example.com:25

# Test 3: Empty MAIL FROM
swaks --to victim@elsewhere.com --from "" \
      --server mail.example.com:25

# Test 4: Percent-hack (legacy)
swaks --to "victim%elsewhere.com@example.com" \
      --from attacker@external.test \
      --server mail.example.com:25

# Test 5: Bracketed-route (legacy)
swaks --to "@example.com:victim@elsewhere.com" \
      --from attacker@external.test \
      --server mail.example.com:25

# Test 6: Source-routed
swaks --to "victim@elsewhere.com@example.com" \
      --from attacker@external.test \
      --server mail.example.com:25

# Test 7: IP-literal destination
swaks --to "victim@[1.2.3.4]" \
      --from attacker@external.test \
      --server mail.example.com:25

Expected response for all seven:

554 5.7.1 Relay access denied

If any of them returns 250 OK, you have an open relay. Stop reading and fix it now.

4. Auth surface test

Verify SASL is only available over TLS:

# Plain connection — should NOT advertise AUTH
swaks --server mail.example.com:587 --quit-after FIRST-EHLO

# After STARTTLS — SHOULD advertise AUTH
swaks --server mail.example.com:587 -tls --quit-after EHLO

Look at the

EHLO

250-AUTH PLAIN LOGIN

Then test rate limiting. From an external IP:

for i in {1..20}; do
    swaks --server mail.example.com:587 -tls \
          --auth-user fake@example.com \
          --auth-password wrongpassword$i \
          --quit-after AUTH 2>&1 | grep -E "535|421"
done

After 5–10 failures, you should see

421

smtpd_client_auth_rate_limit

Repeat for IMAP:

for i in {1..20}; do
    nc -w 2 mail.example.com 993 < /dev/null 2>&1 | head -1
done

After 5–10 connections from the same IP within a minute, Dovecot's anvil should start refusing.

5. SMTP smuggling probe

Test for the December 2023 SMTP smuggling vulnerability. The probe:

# Use printf — `echo -e` behavior depends on which shell and which echo
# implementation runs the script, and the bytes here have to be exact.
( printf 'EHLO probe.test\r\n'
  sleep 1
  printf 'MAIL FROM:<probe@external.test>\r\n'
  sleep 1
  printf 'RCPT TO:<postmaster@example.com>\r\n'
  sleep 1
  printf 'DATA\r\n'
  sleep 1
  printf 'Subject: smuggling probe\r\n\r\n'
  printf 'First message body\r\n'
  printf '\n.\n'                                     # <-- bare LF dot bare LF
  printf 'MAIL FROM:<spoofed@example.com>\r\n'
  printf 'RCPT TO:<postmaster@example.com>\r\n'
  printf 'DATA\r\n'
  printf 'Subject: smuggled\r\n\r\n'
  printf 'If you receive this, smuggling works\r\n'
  printf '\r\n.\r\n'
  printf 'QUIT\r\n'
) | nc mail.example.com 25

What to look for: A vulnerable server will respond

250 OK

501

MAIL FROM

Postfix's mitigation is

smtpd_forbid_bare_newline = yes

postconf smtpd_forbid_bare_newline

If it's

no

6. STARTTLS stripping check

Verify your server enforces TLS where it should. On port 587 with

smtpd_tls_security_level = encrypt

swaks --server mail.example.com:587 \
      --to me@example.com --from sender@external.test \
      --quit-after RCPT

Without

-tls

530 5.7.0 Must issue a STARTTLS command first

MAIL FROM

RCPT TO

For port 25 (server-to-server), TLS should be opportunistic — a peer without TLS support should still get mail. But your outbound TLS configuration should validate certificates properly. Test with:

postconf smtp_tls_security_level smtp_tls_CAfile

For domains with MTA-STS or DANE, set

smtp_tls_security_level = dane

smtp_tls_policy_maps

7. SPF, DKIM, DMARC verification

# SPF
dig +short TXT example.com
# Look for: v=spf1 ... -all (not ~all, not +all)

# DKIM (replace selector if different)
dig +short TXT default._domainkey.example.com

# DMARC
dig +short TXT _dmarc.example.com
# Look for: p=quarantine or p=reject (not p=none in production)

Then send a message to

mail-tester.com

• SPF valid and aligned
• DKIM valid and aligned
• DMARC policy
• Reverse DNS configured
• Server not on blocklists

A score below 9/10 needs investigation.

For DKIM specifically, verify the key is reachable and parseable:

opendkim-testkey -d example.com -s default -vvv

key OK

8. Header-injection probes against web→mail bridges

If your server hosts any web application that sends mail (contact form, password reset, signup confirmation), the web layer is part of your mail attack surface. Test it:

# CR/LF injection in a form field
curl -X POST https://example.com/contact \
     -d "email=attacker@evil.com%0d%0aBcc:%20victim@elsewhere.com" \
     -d "message=hello"

Then check whether

victim@elsewhere.com

The fix lives in the web application, not in Postfix. But you can layer a defense at the mail server with

header_checks

# /etc/postfix/header_checks
/^Bcc:/  IGNORE
/^X-Original-To:/  IGNORE
/^Received:.*by\s+localhost/  IGNORE

…depending on what your application legitimately sends. The fix is in the app code (use a library, validate inputs).

9. Authentication backend check

If you're using SQL or LDAP for auth, verify:

• The Dovecot service account has minimum privileges (read-only on the user table)
• Passwords are stored as bcrypt or argon2, not MD5 or SHA1
• The connection from Dovecot to LDAP/SQL uses TLS
• Failed auth attempts log enough detail for fail2ban to act

Specifically, check:

doveconf -P passdb_args  # Should NOT show plain passwords
grep -E "scheme=" /etc/dovecot/conf.d/auth-sql.conf.ext
# Look for: default_pass_scheme = BLF-CRYPT or ARGON2ID

Bcrypt (

BLF-CRYPT

ARGON2ID

10. Logging and alerting verification

This isn't a probe, it's a sanity check. Trigger known events and verify they appear in the log:

# Trigger a failed auth
swaks --server mail.example.com:587 -tls \
      --auth-user nobody@example.com --auth-password wrong

# Check the log
sudo grep "authentication failure" /var/log/mail.log | tail -5

If the failure isn't logged with the source IP and the username attempted, your

auth_verbose = yes

The 30-item checklist

Print this out, run it before every major mail-server change, and after every significant patch:

Recon (1–4)

1. ☐
   Copy

   nmap -sV -p 25,110,143,465,587,993,995
   returns expected services only
2. ☐ Banner does not leak Postfix/Dovecot version
3. ☐ Reverse DNS (PTR) matches forward DNS for the IP
4. ☐ No unexpected services on adjacent ports

TLS (5–10)

1. ☐
   Copy

   testssl.sh
   shows no SSLv2/v3/TLSv1.0/v1.1 on any mail port
2. ☐ No weak ciphers (RC4, 3DES, EXPORT, NULL) on any mail port
3. ☐ Certificate is valid, not expired, matches hostname
4. ☐ DH parameters ≥ 2048 bits
5. ☐ Submission port 587 requires STARTTLS before AUTH
6. ☐ Port 465 (SMTPS) is enabled with
   Copy

   smtpd_tls_wrappermode = yes

Relay and recipient policy (11–15)

1. ☐ All seven open-relay tests return 5xx rejection
2. ☐
   Copy

   smtpd_relay_restrictions
   exists and contains relay logic
3. ☐
   Copy

   mynetworks
   is not set to
   Copy

   0.0.0.0/0
   or a wide private range you don't control
4. ☐ Rejection responses are clear (no leaking internal hostnames or paths)
5. ☐ Recipient verification rejects non-existent local addresses

Authentication (16–20)

1. ☐ Dovecot
   Copy

   disable_plaintext_auth = yes
2. ☐ Auth mechanisms over plaintext (PLAIN, LOGIN) are rejected
3. ☐ Brute-force rate limiting is enforced (Postfix anvil + Dovecot + fail2ban)
4. ☐ Password storage uses bcrypt or Argon2id
5. ☐ No master accounts left over from migration tools

Smuggling and protocol (21–24)

1. ☐
   Copy

   smtpd_forbid_bare_newline = yes
2. ☐ Smuggling probe rejects the bare-LF terminator
3. ☐ STARTTLS upgrade discards pre-handshake buffer (test by injection probe)
4. ☐ No NOOP/RSET responses leak server state

DNS and identity (25–27)

1. ☐ SPF record exists, is
   Copy

   -all
   (or
   Copy

   ~all
   only if intentional)
2. ☐ DKIM key publishes correctly and
   Copy

   opendkim-testkey
   returns OK
3. ☐ DMARC policy is
   Copy

   p=quarantine
   or
   Copy

   p=reject
   (not
   Copy

   p=none
   )

Operations (28–30)

1. ☐ Mail logs ship to a separate host
2. ☐ Alerting fires on auth-failure spikes, queue growth, and outbound volume anomalies
3. ☐ Patches applied within 7 days of upstream security release

What attackers will do that this checklist doesn't catch

Two categories of attack the mail-layer checklist doesn't cover, by design:

Credential phishing. No mail-server hardening helps if your users type their passwords into fake login pages. Email security at this layer is about your server, not your users. User training and 2FA are separate concerns.

Application-layer abuse. Once an attacker has valid credentials, your IMAP server happily lets them search and download every message in the account. There's no "this attacker authenticated correctly but seems suspicious" detection at the IMAP layer in any major server. Logging anomalies after the fact is the best you can do.

For both categories: 2FA on the mail accounts (Dovecot supports OAUTHBEARER and various 2FA bridges), monitoring for impossible-travel logins, and incident-response procedures matter as much as anything in this checklist.

Run this every six months

Mail server configurations drift. New tutorials get published with bad advice. Distribution upgrades change defaults. New CVEs land. The 30-item checklist is not a one-time exercise — it's a recurring audit. Put it on your calendar twice a year. The work takes about an hour. The cost of skipping it is whatever an open relay or stripped-TLS submission portal costs your domain's reputation.

What I'd Tell My Past Self

The first time I pen-tested a mail server I'd configured myself, I found two of these issues. Both had been there for months. Neither had been exploited yet, but neither would have been visible to me from the inside — both were "your config is technically correct except for this one ordering issue" problems that only show up when you're attacking from the outside.

The lesson generalises: you cannot audit your own configuration by reading it. You have to attack it. Static review will tell you whether the syntax is correct; dynamic probing will tell you whether the behavior is correct. Both matter, but the second one is what catches the bugs that ship.

That ends the email security series. Posts 1 through 4 gave you the mental model. Posts 5 through 7 gave you the failure modes. This post gave you the toolkit. The next mail server you set up — yours, your employer's, or a friend's you're helping — should be measurably more secure than the last one. That's the whole point.