Deploy Self-Hosted Email Solution

Deploy Self-Hosted Email Solution

CompactHost ·

Running your own email server gives you complete control over your data, eliminates reliance on third-party providers, and lets you use custom domains without paying per-user subscriptions. I've deployed Mailcow on a small VPS, and it's transformed how I think about email sovereignty. In this guide, I'll walk you through deploying a production-ready self-hosted email solution from scratch.

Why Self-Host Email?

When you use Gmail or Outlook, you're trusting your messages to corporations who scan your content and profile your behavior. Self-hosting email means you own the entire stack: the mail server, the webmail interface, the spam filters, even the backup strategy. I prefer this model because it aligns with my philosophy of digital independence.

The downside? You need stable infrastructure. Self-hosted email requires a static IP, reliable uptime, proper DNS configuration, and careful attention to spam reputation. If you're running on residential internet or a flaky VPS, your mail won't reach inboxes. For most homelabbers, I recommend a dedicated VPS from a provider like RackNerd (which offers affordable, reliable KVM VPS starting at $10/year) rather than self-hosting from home.

Choosing Your Email Stack

I've tested both Mailcow and Mailu, and they're the two most viable self-hosted email systems for homelab enthusiasts. Mailcow runs on Docker, offers a powerful admin panel, and includes built-in support for SOGo (webmail), Roundcube, and mobile sync. Mailu is lighter weight but slightly less polished. I prefer Mailcow because its UI is cleaner and it handles certificate renewal automatically.

Both assume you have:

Prerequisites: DNS and Domain Setup

Before you deploy, your DNS records must be correct. Email delivery relies entirely on DNS. You'll need:

Get your reverse DNS set up first. This is the biggest gotcha I've seen. Your mail server's IP must have a PTR record pointing back to a hostname that resolves to that IP. Without it, major providers (Gmail, Outlook, Yahoo) will reject your mail. Contact your VPS provider and request PTR setup for your IP.

Deploying Mailcow with Docker Compose

I'm using Mailcow because it's the easiest to get working. Here's the deployment process:

Tip: SSH into your VPS as root or with sudo access. Make sure UFW or your firewall allows ports 25, 80, 443, 465, 587, 993, and 995 inbound.

First, clone the Mailcow repository and generate configuration:

cd /opt
git clone https://github.com/mailcow/mailcow-dockerized
cd mailcow-dockerized
bash generate_config.sh

The script will ask for your domain (e.g., example.com), hostname (e.g., mail.example.com, and timezone. Answer carefully—changing these later is a pain.

Next, review and edit the generated .env file. Key settings I always check:

# Edit the configuration
nano .env

# Key variables to verify:
# MAILCOW_HOSTNAME=mail.example.com
# DOMAIN=example.com
# SKIP_CLAMD=y (unless you have 8GB+ RAM; ClamAV is heavy)
# SKIP_SOLR=y (Solr is optional; I disable it for smaller deployments)
# SKIP_IP_CHECK=n (leave as-is unless you're behind NAT)
# TZ=UTC (adjust to your timezone)

Now pull the Docker images and start the stack:

# Pull all images (this takes a few minutes)
docker compose pull

# Start the stack in the background
docker compose up -d

# Watch logs to ensure everything started
docker compose logs -f

Wait 2–3 minutes for Mailcow to initialize. You'll see a lot of database setup messages. When you see something like mailcow-dockerized-mysql-1 | Ready for start up, the containers are ready.

Access the admin panel at https://mail.example.com/admin. Default login is admin with password moohoo (you must change this immediately). The web interface will guide you through adding domains, users, and aliases.

Watch out: Mailcow serves HTTPS using Let's Encrypt. If your domain DNS isn't pointing to your server yet, certificate generation will fail. Ensure your A record is live before starting Docker Compose.

Configuring DNS Records for Email Delivery

Now that Mailcow is running, you need to add DNS records so other mail servers can find and validate your email:

1. MX Record

example.com MX 10 mail.example.com.

2. SPF Record (TXT)

example.com TXT "v=spf1 ip4:YOUR.SERVER.IP mx ~all"

Replace YOUR.SERVER.IP with your actual public IP address. This tells receivers: "Mail for my domain comes from this IP or from my MX server."

3. DKIM Record

Log into Mailcow's admin panel → Domains → your domain → DKIM keys. Click "Generate DKIM keys" and copy the long string. Add a TXT record:

dkim._domainkey.example.com TXT "v=DKIM1; k=rsa; p=VERY_LONG_PUBLIC_KEY_HERE"

4. DMARC Record

_dmarc.example.com TXT "v=DMARC1; p=quarantine; rua=mailto:[email protected]"

This tells receivers to quarantine (hold) emails that fail SPF/DKIM validation and send reports to your admin address.

After adding these records, wait 15–30 minutes for propagation, then test at MXToolbox or DMARCian.

Creating Users and Testing Email

In the Mailcow admin panel, add a mailbox:

Once created, log into the webmail at https://mail.example.com using your new address and password. Send yourself a test email from your main account (Gmail, etc.) to confirm delivery works.

To test outbound mail from your server, send an email to [email protected] and check the score. A good score is 8/10 or higher. Common issues:

Mobile and Desktop Client Setup

To use Mailcow with Thunderbird, Apple Mail, or mobile clients, configure IMAP/SMTP:

Most modern clients auto-detect these settings. If not, configure manually.

Maintenance and Backups

Self-hosted email requires ongoing care. Set up automated backups by adding this to your crontab:

# Backup Mailcow daily at 2 AM
0 2 * * * cd /opt/mailcow-dockerized && docker compose exec -T mysql mysqldump -u mailcow -p$MYSQL_PASSWORD mailcow > /backups/mailcow-$(date +\%Y\%m\%d).sql

Also keep your Docker images updated. I run this monthly:

cd /opt/mailcow-dockerized
docker compose pull
docker compose up -d

Monitor logs regularly for errors:

docker compose logs postfix | tail -50
docker compose logs dovecot | tail -50

Firewall Rules

Use UFW to restrict access. Allow only required ports:

ufw default deny incoming
ufw allow 22/tcp
ufw allow 25/tcp
ufw allow 80/tcp
ufw allow 143/tcp
ufw allow 443/tcp
ufw allow 465/tcp
ufw allow 587/tcp
ufw allow 993/tcp
ufw allow 995/tcp
ufw enable

Port 25 must be open to the world so other mail servers can deliver mail to you. Ports 143, 465, 587, 993, 995 can be restricted to your IP range if you're only connecting from home, but for flexibility, I leave them open.

Next Steps

You now have a working self-hosted email server. Next, consider:

Self-hosted email is demanding but rewarding. You own your data, control your reputation, and gain deep knowledge of how email actually works. If you're serious about digital sovereignty, this is non-negotiable infrastructure.

```