Hardening a Self-Hosted Mail Server: fail2ban, Postfix, and Dovecot

My home server runs a full mail stack — Postfix, Dovecot, Rspamd, ClamAV — on MicroShift. It’s exposed directly to the internet on ports 25 and 587. That means it gets attacked. Here’s how I protect it, what the attacks look like in the logs, and how I recently tightened the configuration after spotting a gap.

The Threat Landscape

Two distinct attack types show up regularly in the logs:

1. SASL Brute-Force — Attackers try username/password combinations via SMTP AUTH or IMAP. The pattern: multiple IPs, each trying a different username, typically in parallel waves.

2. Recipient Enumeration (Directory Harvest) — Attackers connect via SMTP and send RCPT TO commands for dozens of addresses to discover which ones exist. They don’t need credentials — Postfix helpfully responds with 550 5.1.1 User unknown for every non-existent address.

The Defense Stack

fail2ban on the Host

fail2ban runs directly on the Fedora host (not inside any container). The mail pods run with hostNetwork: true — without that, OVN-Kubernetes would SNAT all external connections to its own internal gateway IP, making it impossible for Postfix and Dovecot to see real client IPs.

All bans use firewallcmd-rich-rules[actiontype=<allports>] — a banned IP can’t reach any port on the server, not just mail ports.

Jail Architecture

The key insight is having two levels of response for each protocol: a lenient catch-all jail and a strict “unknown user” jail.

The -unknown jails fire immediately because there is no legitimate reason for an unknown username to appear. A real user who misremembers their password will still use a valid username.

How Fast Does This Work?

For the SASL attack on May 1st, checking the fail2ban log against the Postfix log shows the ban happening within 1–2 seconds of the first failed attempt:

06:43:08 – postfix-sasl-unknown: Found 182.95.120.50
06:43:08 – postfix-sasl-unknown: Ban 182.95.120.50   ← same second

Every IP in that attack wave was banned on its first attempt, before it could try a second.

postfix-sasl-unknown in Detail

The filter matches the sasl_username= field that Postfix logs only when the client actually sent credentials:

failregex = warning: \S+\[<HOST>\](?::\d+)?: SASL \S+ authentication failed: .*, sasl_username=<F-USER>\S+</F-USER>

But a script (/usr/local/bin/fail2ban-sasl-mysql-check.sh) runs as ignorecommand before banning: it looks up the attempted username in the Postfix MySQL database. If the user does exist (wrong password, not a harvest), the IP is ignored by this jail — the regular postfix-sasl jail with its 3-attempt threshold handles that case instead.

dovecot-unknown in Detail

Dovecot’s auth-worker logs sql: unknown user when a lookup against the database returns nothing. That’s unambiguous — no ignorecommand needed:

failregex = auth-worker\(<F-USER>[^,]+</F-USER>,<HOST>\)(?:<[^>]+>)*: request \[\d+\]: Info: sql: unknown user

The Gap: Recipient Enumeration

The SASL and IMAP jails worked well. But analysing the logs I found that recipient enumeration attacks had a different character — IP 122.185.178.134 tried 20 non-existent recipients in a single SMTP session, all within 2 seconds:

09:45:32 – connect from unknown[122.185.178.134]
09:45:34 – reject: RCPT from unknown[122.185.178.134]: 550 5.1.1 <42hcwh5m...@michaelkoester.de>: User unknown
09:45:34 – reject: RCPT from unknown[122.185.178.134]: 550 5.1.1 <wn11c3vk...@michaelkoester.de>: User unknown
... (18 more) ...
09:45:34 – NOQUEUE: too many errors after RCPT from unknown[122.185.178.134]
09:45:34 – disconnect from unknown[122.185.178.134] ehlo=1 mail=1 rcpt=0/20

Postfix uses smtpd_hard_error_limit (default: 20) to decide when to disconnect. fail2ban only sees the log entries after the session ends — by then all 20 probes are complete.

Two-Layer Fix

Layer 1 — Postfix limits the session:

smtpd_hard_error_limit = 5    # disconnect after 5 errors (was 20)
smtpd_soft_error_limit = 3    # start delaying responses after 3 errors (was 10)

This goes in the Postfix ConfigMap. After a rolling restart of the pod the values are live:

$ kubectl exec -n mailstack deploy/mail-postfix -- postconf smtpd_hard_error_limit smtpd_soft_error_limit
smtpd_hard_error_limit = 5
smtpd_soft_error_limit = 3

Layer 2 — fail2ban bans the IP after 3 rejections:

New filter /etc/fail2ban/filter.d/postfix-rcpt-unknown.conf:

[Definition]
failregex = reject: RCPT from \S+\[<HOST>\][^:]*: 550 5\.1\.1 \S+: Recipient address rejected: User unknown in virtual mailbox table

datepattern = %%b %%d %%H:%%M:%%S
              {^LN-BEG}

New jail in /etc/fail2ban/jail.local:

[postfix-rcpt-unknown]
enabled        = true
filter         = postfix-rcpt-unknown
action         = firewallcmd-rich-rules[actiontype=<allports>]
backend        = polling
logpath        = /var/lib/pvc/pvc-5575476b-0dfb-498f-b979-4f9d9de2ccae_mailstack_mail-logs/postfix.log
maxretry       = 3
bantime        = 604800
findtime       = 3600

maxretry=3 rather than 1 to avoid banning legitimate senders who happen to mistype a recipient — a single typo is human, three “User unknown” responses in an hour is an attack.

Testing the filter against the live log confirmed 379 matches before enabling:

$ fail2ban-regex postfix.log /etc/fail2ban/filter.d/postfix-rcpt-unknown.conf
Failregex: 379 total

Result

With both layers active, a directory harvest attack now plays out like this:

1. Attacker connects, tries recipient 1 → 550 User unknown (fail2ban: Found 1/3)
2. Tries recipient 2 → 550 User unknown (fail2ban: Found 2/3)
3. Tries recipient 3 → 550 User unknown (fail2ban: Found 3/3 → Ban, all ports, 7 days)
4. Tries recipient 4 → Postfix starts delaying responses (smtpd_soft_error_limit)
5. Tries recipient 5 → Connection terminated (smtpd_hard_error_limit)

Even in the worst case (fail2ban polls just after the session ends), the attacker gets at most 5 probes per session and is then locked out for a week.