Configure Personal Mail Server

Configure Personal Mail Server

CompactHost ·

I've been running my own mail server for three years now, and it's one of those projects that pays dividends every single day. No corporate email provider reading your inbox, no surprise policy changes, total control over your data. But I'll be honest: mail is *hard*. DNS records, DKIM, SPF, DMARC—it's a steep curve. That said, once you get it working, the satisfaction is real, and the cost is negligible.

This guide will walk you through setting up a functional mail server on a minimal VPS—I recommend a RackNerd KVM VPS (they're cheap, reliable, and their IPs aren't blacklisted out of the gate). We'll use Postfix, Dovecot, and SpamAssassin to create something production-ready.

Prerequisites and Server Specs

You'll need a few things before we start:

I prefer Debian 12 for mail servers—clean, stable, good package versions. Ubuntu 22.04 LTS works too. Avoid shared hosting; mail servers need their own IP address and proper reverse DNS.

Tip: Set up reverse DNS (PTR record) for your server IP before anything else. Ask your provider to do this. Many ISPs and hosting companies will reject mail from IPs with no reverse DNS, and your deliverability will suffer.

Install Core Mail Packages

Start with a fresh server. Update everything, then install Postfix (SMTP), Dovecot (IMAP/POP3), and SpamAssassin:

sudo apt update && sudo apt upgrade -y
sudo apt install -y postfix postfix-mysql dovecot-core dovecot-imapd dovecot-pop3d dovecot-mysql spamassassin spamc opendkim opendkim-tools fail2ban ufw certbot python3-certbot-nginx

During the Postfix installation, you'll get an interactive menu. Choose "Internet Site" and set your mail server hostname (e.g., mail.yourdomain.com). We'll fine-tune this in config files anyway.

Obtain an SSL Certificate

Mail clients require TLS. Let's get a free certificate from Let's Encrypt:

sudo certbot certonly --standalone -d mail.yourdomain.com -d yourdomain.com

Certbot will place certificates in /etc/letsencrypt/live/mail.yourdomain.com/. Note these paths; we'll reference them soon.

Create a renewal hook so Dovecot and Postfix reload after renewal:

sudo mkdir -p /etc/letsencrypt/renewal-hooks/post
sudo nano /etc/letsencrypt/renewal-hooks/post/mail.sh

Paste:

#!/bin/bash
systemctl reload postfix
systemctl reload dovecot

Then make it executable:

sudo chmod +x /etc/letsencrypt/renewal-hooks/post/mail.sh

Configure Postfix (SMTP)

Postfix is your mail router. Back up the original config, then replace it:

sudo cp /etc/postfix/main.cf /etc/postfix/main.cf.backup
sudo nano /etc/postfix/main.cf

Replace the entire file with this:

myhostname = mail.yourdomain.com
myorigin = yourdomain.com
inet_interfaces = all
inet_protocols = ipv4
mydestination = $myhostname, yourdomain.com, localhost.localdomain, localhost
mynetworks = 127.0.0.0/8 [::1]/128
relay_domains =

# TLS Settings
smtpd_tls_cert_file = /etc/letsencrypt/live/mail.yourdomain.com/fullchain.pem
smtpd_tls_key_file = /etc/letsencrypt/live/mail.yourdomain.com/privkey.pem
smtpd_tls_security_level = may
smtp_tls_security_level = may
smtpd_tls_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1
smtp_tls_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1

# SASL Authentication
smtpd_sasl_type = dovecot
smtpd_sasl_path = private/auth
smtpd_sasl_auth_enable = yes
smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination

# Virtual Users (optional, but recommended)
virtual_mailbox_domains = yourdomain.com
virtual_mailbox_base = /var/mail/vhosts
virtual_mailbox_maps = hash:/etc/postfix/vmailbox
virtual_alias_maps = hash:/etc/postfix/virtual
virtual_uid_maps = static:5000
virtual_gid_maps = static:5000

# DKIM
milter_default_action = accept
milter_protocol = 6
smtpd_milters = inet:localhost:12301
non_smtpd_milters = inet:localhost:12301

Replace yourdomain.com and mail.yourdomain.com with your actual domain.

Now create the virtual user database if you want multiple email addresses:

sudo mkdir -p /var/mail/vhosts/yourdomain.com
sudo groupadd -g 5000 vmail
sudo useradd -m -d /var/mail/vhosts -s /usr/sbin/nologin -u 5000 -g vmail vmail
sudo chown -R vmail:vmail /var/mail/vhosts

Create the virtual mailbox map:

sudo nano /etc/postfix/vmailbox

Add your users (one per line):

[email protected] yourdomain.com/admin/
[email protected] yourdomain.com/info/

Then hash the database:

sudo postmap /etc/postfix/vmailbox

Test Postfix config and reload:

sudo postfix check
sudo systemctl restart postfix
Watch out: If Postfix won't start, check logs with sudo journalctl -u postfix -n 50. Common issues: typos in paths, permission problems on cert files, or firewall blocking port 25.

Configure Dovecot (IMAP/POP3)

Dovecot handles email retrieval. Edit the main config:

sudo nano /etc/dovecot/dovecot.conf

Ensure these lines are present (uncomment if needed):

listen = *, ::
protocols = imap pop3

ssl = required
ssl_cert = </etc/letsencrypt/live/mail.yourdomain.com/fullchain.pem
ssl_key = </etc/letsencrypt/live/mail.yourdomain.com/privkey.pem

mail_location = maildir:/var/mail/vhosts/%d/%n
mail_uid = vmail
mail_gid = vmail

service auth {
  unix_listener /var/spool/postfix/private/auth {
    mode = 0666
    user = postfix
    group = postfix
  }
}

userdb {
  driver = static
  args = uid=vmail gid=vmail home=/var/mail/vhosts/%d/%n
}

passdb {
  driver = static
  args = password=test
}

This uses static authentication (for testing). For production, switch to a database or system users. But for a small personal server, this works fine. Replace "test" with a strong password:

sudo systemctl restart dovecot

Set Up DNS Records

Mail depends entirely on DNS. Add these records to your domain's DNS provider:

Generate DKIM keys:

sudo opendkim-genkey -b 2048 -d yourdomain.com -D /etc/opendkim/keys/yourdomain.com -s default -v
sudo chown opendkim:opendkim /etc/opendkim/keys/yourdomain.com/default.private
sudo cat /etc/opendkim/keys/yourdomain.com/default.txt

Copy the output (between quotes) into a DKIM TXT record in your DNS. It'll look like:

default._domainkey.yourdomain.com TXT "v=DKIM1; h=sha256; k=rsa; p=YOUR_PUBLIC_KEY..."

Wait 24 hours for DNS to propagate, then test:

dig yourdomain.com MX
dig default._domainkey.yourdomain.com TXT

Enable Firewall and Fail2Ban

Secure your server:

sudo ufw allow 22/tcp
sudo ufw allow 25/tcp
sudo ufw allow 143/tcp
sudo ufw allow 110/tcp
sudo ufw allow 465/tcp
sudo ufw allow 587/tcp
sudo ufw enable

Enable Fail2Ban to rate-limit auth attempts:

sudo nano /etc/fail2ban/jail.local

Add:

[dovecot-auth]
enabled = true
port = imap,pop3
logpath = /var/log/mail.log
maxretry = 5
findtime = 600
bantime = 3600

[postfix-smtp]
enabled = true
port = smtp
logpath = /var/log/mail.log
maxretry = 10

Reload Fail2Ban:

sudo systemctl restart fail2ban

Test Your Mail Server

Send a test email using telnet or mail clients. I recommend Thunderbird for testing—it auto-discovers settings. Add an account with:

Use online mail tester tools like Mail Tester or MXToolbox to verify SPF, DKIM, and DMARC alignment.

Next Steps

Once you've got mail flowing, consider adding:

Running your own mail server means you own your digital identity. It takes effort upfront, but you'll never again worry about a provider shutting down your account or selling your metadata. Start small, test thoroughly, and you'll have a mail setup that'll last years.