Common Homelab Networking Mistakes and How to Avoid Them

Common Homelab Networking Mistakes and How to Avoid Them

CompactHost ·

I've seen countless homelab setups fail silently—not because the hardware was bad, but because the networking foundation was fundamentally broken. Most people jump into self-hosting without thinking about network isolation, firewall rules, or VLAN segmentation. Then they wonder why a compromised service became a backdoor into their entire network.

In this article, I'm sharing the networking mistakes I've made and observed in other homelabs, along with practical fixes you can implement today. These aren't theoretical; they're lessons learned from real infrastructure.

Mistake 1: Running Everything on a Flat Network

This is the cardinal sin of homelab networking. A flat network means every device can reach every other device without any barriers. You run your Nextcloud instance, your Jellyfin server, your DNS resolver, and your IoT camera all on the same subnet with no segmentation.

When—not if—one service gets compromised, an attacker has free rein across your entire infrastructure. They can pivot to your database servers, steal files from your NAS, or use your homelab as a pivot point to attack other networks.

I made this mistake in my first year of self-hosting. I had Nextcloud on 192.168.1.50, Jellyfin on 192.168.1.51, and my NAS on 192.168.1.20, all chatting freely. When I eventually implemented VLANs and firewall rules, I realized just how exposed I'd been.

The Fix: Implement network segmentation using VLANs. I use Ubiquiti UniFi for this—it's affordable and well-documented—but you can also segment with managed switches or even Docker network policies on your hypervisor.

Create separate VLANs for:

Set firewall rules between VLANs so services can only communicate what they actually need. For example, your web services can talk to storage, but guest devices cannot.

Mistake 2: Neglecting Inbound Firewall Rules

You set up a reverse proxy (Caddy, Nginx, Traefik) to expose your services to the internet, then you assume you're done with firewall rules. Wrong.

Your homelab machine is still listening on ports 22 (SSH), 3000 (monitoring), 5432 (database), and a dozen other services on your LAN IP. If someone gets your home IP and starts port-scanning, they'll find these open ports. An attacker doesn't need to compromise your reverse proxy; they can just brute-force SSH or exploit an unpatched database listener.

I learned this the hard way when I found failed SSH login attempts from VPS providers in my auth logs—hundreds of them. Someone was methodically scanning residential IP blocks.

The Fix: Use a stateful firewall with explicit inbound rules. I recommend UFW on Linux or hardware firewalls like Ubiquiti Dream Machine.

# On your main homelab host, enable UFW and set defaults
sudo ufw default deny incoming
sudo ufw default allow outgoing

# Allow SSH only from your trusted networks
sudo ufw allow from 192.168.1.0/24 to any port 22
sudo ufw allow from 10.8.0.0/24 to any port 22 # VPN subnet

# Allow HTTPS inbound (for your reverse proxy)
sudo ufw allow 443/tcp
sudo ufw allow 80/tcp

# Explicitly deny internal service ports from the internet
sudo ufw deny 3306/tcp  # MySQL
sudo ufw deny 5432/tcp  # PostgreSQL
sudo ufw deny 6379/tcp  # Redis
sudo ufw deny 9200/tcp  # Elasticsearch

# Enable the firewall
sudo ufw enable

# Verify rules
sudo ufw status verbose
Watch out: Don't lock yourself out. Test SSH access from your expected networks before enabling the firewall. If you do get locked out, you'll need physical access or a serial console to recover.

Mistake 3: Ignoring Upstream DNS and DHCP Security

Your router assigns IP addresses via DHCP and forwards DNS queries upstream. Most people never touch these settings. That's a problem.

Without DHCP snooping or a configured DNS firewall, any device on your network (or a guest who connects to your WiFi) can become a DHCP server and hijack your network. They can hand out their IP as the default gateway and intercept all traffic. Similarly, unprotected DNS resolvers allow DNS spoofing—redirecting traffic to malicious sites.

I always set up Pi-hole or Adguard Home as my recursive DNS resolver, configured on my router so all DHCP clients use it by default. This gives me visibility into DNS queries, blocks known malware domains, and prevents external DNS exfiltration.

The Fix: Deploy a local DNS resolver (Pi-hole, Adguard Home, or Unbound) and point all DHCP clients to it. Then configure your router or UniFi controller to:

Mistake 4: Weak or Default Credentials on Network Devices

This sounds obvious, but I've walked into so many homelabs where the router, switch, and NAS all had default admin passwords. "admin / admin" or "admin / password" is still shockingly common.

Network devices are often overlooked in security audits. You harden your Nextcloud and SSH servers, but leave your UniFi controller running with the factory default password. An attacker who breaches your Nextcloud can pivot to the controller, change VLAN rules, and suddenly they have free access to your storage VLAN.

The Fix: Change every default credential, immediately:

Use a password manager and generate strong, unique passwords. Enable MFA on any device that supports it. For UniFi, I enable the "Advanced" authentication settings and require SSH key access for API calls.

Mistake 5: No Rate Limiting or Brute-Force Protection

Your Vaultwarden or Nextcloud is exposed through a reverse proxy. Someone runs a password spray attack—slowly, across many IPs—and tries thousands of common passwords. Without rate limiting, they might get in.

I've seen homelabs where the reverse proxy was hardened but the backend application wasn't. They added Fail2ban to block SSH, but forgot to rate-limit login attempts on their web services. An attacker can make 100 requests per second to `/login` and eventually crack credentials.

The Fix: Implement rate limiting at two layers:

Layer 1 — Reverse Proxy: Caddy (my preference) has built-in rate limiting:

example.com {
    rate_limit / 10r/s
    
    reverse_proxy localhost:8080 {
        # Health check and timeouts
        health_uri /health
        health_timeout 5s
    }
}

Layer 2 — Application: Most self-hosted apps support login throttling. In Vaultwarden, set:

For SSH, use Fail2ban to automatically block IPs with repeated failed attempts:

# Install and enable
sudo apt install fail2ban
sudo systemctl enable fail2ban

# Edit /etc/fail2ban/jail.local
[sshd]
enabled = true
port = 22
filter = sshd
logpath = /var/log/auth.log
maxretry = 3
findtime = 600
bantime = 3600

sudo systemctl restart fail2ban

Mistake 6: Not Using VPN or Reverse Proxy for Remote Access

You want to access your Nextcloud from your phone while away from home. The easy solution is to port-forward 443 to your homelab machine and access it directly. The wrong solution.

Now your Nextcloud is directly exposed to the internet with no additional layer of authentication or monitoring. Every vulnerability in Nextcloud is a direct path into your network. Every port scanner on the internet will find you.

The right approach: use a reverse proxy with authentication (Caddy + Authelia) or a VPN (Tailscale, WireGuard). This way, even if your Nextcloud has a zero-day, an attacker still can't reach it without credentials or VPN access.

The Fix: I prefer Tailscale for remote access because it's effortless—install on your phone, install on your homelab, and you have zero-trust networking immediately. But if you want traditional HTTPS access, use Caddy with a reverse proxy and add an authentication middleware like Authelia for critical services.

Mistake 7: Ignoring Bandwidth and QoS

A single misbehaving service (runaway Jellyfin transcoding, backup going haywire, or a Samba share being hammered) can saturate your home internet and make everything else unusable. Your Zoom calls drop, your browsing crawls, and you have no idea why.

Without QoS (Quality of Service) rules, all traffic is equal. A backup to your NAS can consume 100% of your bandwidth and starve your production traffic.

The Fix: Configure QoS on your router or gateway. I use UniFi, and I set traffic shaping rules:

On Linux, use `tc` (traffic control) for per-interface rules, or use Docker resource limits to cap container bandwidth.

Mistake 8: Not Monitoring Network Traffic

You set up a firewall and forget about it. Months later, you realize a service has been leaking data, or a compromised container has been exfiltrating files. You had no way to know because you weren't monitoring.

Network visibility is crucial. I run Zeek (formerly Bro) or Suricata for intrusion detection, and I log all DNS queries to Pi-hole so I can see if a service is trying to phone home to unexpected domains.

The Fix: Set up basic network monitoring:

Mistake 9: Hardcoding IPs Instead of Using DNS

You set up a Docker Compose stack and hardcode IPs in your environment files. You configure Nextcloud to sync with a storage server at 192.168.10.50. Then your router reboots, DHCP reassigns IPs, and everything breaks.

Or worse, you use hostnames but don't implement local DNS properly, so your services are doing external DNS lookups that fail when your internet goes down.

The Fix: Use DNS for everything, and make it reliable:

In my Pi-hole, I have entries like:

Every service references these hostnames, so I can migrate or change IPs without updating application config.

Putting It All Together

A secure homelab network isn't built in a day, but starting with these fixes will eliminate 90% of the common vulnerabilities:

  1. Segment your network with VLANs
  2. Implement explicit firewall rules
  3. Deploy a local DNS resolver
  4. Change all default credentials
  5. Add rate limiting and brute-force protection
  6. Use a VPN or reverse proxy for remote access