Installing and Configuring a VPS for Self-Hosting: Initial Setup and Hardening
We earn commissions when you shop through the links on this page, at no additional cost to you. Learn more.
You just rented a fresh VPS. It's yours, it's online, and you're staring at a blank terminal prompt thinking "now what?" I've been there. When I first spun up a box at RackNerd for around $40/year, I made every rookie mistake in the book: weak passwords, open ports, default configurations. This tutorial walks you through the exact hardening steps I now do on every new server, keeping your self-hosted applications safe from the moment you log in.
Why Initial Hardening Matters
Your VPS is immediately exposed to the internet. Within hours, automated scanners probe for open SSH ports, weak credentials, and known vulnerabilities. I learned this the hard way when a box I didn't harden got compromised in 48 hours—nothing catastrophic, but enough to wipe the system and start over. The security footprint is largest on day one, which is why the first steps matter most.
This guide assumes you have SSH access to a fresh Ubuntu 24.04 or Debian 12 installation. If you're on a different distribution, the package names might differ, but the principles remain identical.
Step 1: Generate SSH Keys Locally and Disable Password Auth
The biggest attack surface on any VPS is SSH. Every second, bots hammer port 22 with username/password combinations. I refuse to use password authentication anymore. Instead, I generate a strong ED25519 keypair on my local machine and disable password auth entirely.
On your local machine (not the VPS):
ssh-keygen -t ed25519 -C "[email protected]" -f ~/.ssh/vps_key
# Press Enter for no passphrase (or set one for extra security)
This creates two files: ~/.ssh/vps_key (private) and ~/.ssh/vps_key.pub (public). Keep the private key safe—if it leaks, an attacker can access your VPS.
Now, copy your public key to the VPS. Use your initial password login (usually provided by your VPS provider):
ssh-copy-id -i ~/.ssh/vps_key.pub root@YOUR_VPS_IP_ADDRESS
# You'll be prompted for your password—enter it
# Then test the connection:
ssh -i ~/.ssh/vps_key root@YOUR_VPS_IP_ADDRESS
Once you confirm key-based login works, SSH into the VPS and disable password authentication entirely:
sudo nano /etc/ssh/sshd_config
# Find these lines and ensure they're set to:
PasswordAuthentication no
PubkeyAuthentication yes
PermitRootLogin no
Port 22
# (Change Port to something non-standard like 2222 if you want extra obscurity)
# Save and exit (Ctrl+X, Y, Enter)
# Restart SSH:
sudo systemctl restart ssh
Step 2: Create a Non-Root User
Running everything as root is dangerous. A compromised application or script running as root has full system access. I always create a dedicated user for self-hosted applications.
sudo useradd -m -s /bin/bash selfhost
sudo usermod -aG sudo selfhost
# Add this user to Docker group (we'll install Docker next):
sudo usermod -aG docker selfhost
# Verify:
id selfhost
Now copy your SSH key to this user's home directory:
sudo su - selfhost
mkdir -p ~/.ssh
sudo cat /root/.ssh/authorized_keys > ~/.ssh/authorized_keys
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
# Test login:
exit
ssh -i ~/.ssh/vps_key selfhost@YOUR_VPS_IP_ADDRESS
From now on, use the `selfhost` user for deployments. You can still use `sudo` for administrative tasks.
Step 3: Update System and Install UFW Firewall
Start with a clean slate. Update all packages and install a simple firewall:
sudo apt update && sudo apt upgrade -y
sudo apt install ufw curl wget git -y
# Enable firewall and set defaults:
sudo ufw default deny incoming
sudo ufw default allow outgoing
# Allow SSH (critical—don't lock yourself out):
sudo ufw allow 22/tcp
# Allow HTTP and HTTPS for web services:
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
# Enable the firewall:
sudo ufw enable
# Check the rules:
sudo ufw status
UFW is simple and effective. I prefer it over the more complex iptables or nftables for straightforward VPS setups. As you deploy services, you'll add rules here. For example, if you run a Docker container on port 8080, you'd add: sudo ufw allow 8080/tcp.
Step 4: Install and Configure fail2ban
Even with key-based SSH, attackers still probe. fail2ban monitors logs and temporarily bans IPs after repeated failed login attempts. I've watched it block thousands of attacks from the same IP ranges in a single day.
sudo apt install fail2ban -y
sudo systemctl start fail2ban
sudo systemctl enable fail2ban
# Check status:
sudo fail2ban-client status
# See banned IPs:
sudo fail2ban-client status sshd
The default configuration is good enough for most setups. It bans an IP for 10 minutes after 5 failed SSH attempts. If you want to customize it, create a local override:
sudo nano /etc/fail2ban/jail.local
Add:
[sshd]
enabled = true
port = 22
filter = sshd
logpath = /var/log/auth.log
maxretry = 3
findtime = 600
bantime = 3600
# This bans IPs for 1 hour after 3 failed attempts within 10 minutes
Restart fail2ban to apply changes:
sudo systemctl restart fail2ban
Step 5: Install Docker and Docker Compose
If you're self-hosting applications, Docker is essential. I manage all my services (Nextcloud, Vaultwarden, Gitea, Jellyfin) in containers.
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
# Add your user to the docker group (already done earlier, but verify):
sudo usermod -aG docker selfhost
# Install Docker Compose:
sudo curl -L "https://github.com/docker/compose/releases/download/v2.24.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
# Verify:
docker --version
docker-compose --version
Log out and back in to apply the docker group change:
exit
ssh -i ~/.ssh/vps_key selfhost@YOUR_VPS_IP_ADDRESS
docker ps # Should work without sudo now
Step 6: Enable Automatic Security Updates
Security patches matter. A Zero-day in OpenSSL or a kernel vulnerability can compromise your server. I enable automatic updates so critical patches apply without my intervention:
sudo apt install unattended-upgrades apt-listchanges -y
sudo dpkg-reconfigure -plow unattended-upgrades
# Verify it's running:
sudo systemctl status unattended-upgrades
Check that automatic updates are configured:
sudo nano /etc/apt/apt.conf.d/50unattended-upgrades
# Look for "Unattended-Upgrade::Allowed-Origins" and ensure it includes security updates
Step 7: Configure Swap and Set System Limits
Most cheap VPS tiers come with minimal RAM. A 1GB VPS can run Nextcloud, Vaultwarden, and Jellyfin if you add swap. Swap uses disk as slower memory—not ideal, but it prevents out-of-memory crashes.
# Check current swap:
free -h
# Create 2GB swap file (adjust to your disk space):
sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
# Make it permanent:
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
# Verify:
free -h
Also, increase file descriptor limits for Docker:
sudo nano /etc/security/limits.conf
# Add at the end:
# * soft nofile 65535
# * hard nofile 65535
# * soft nproc 65535
# * hard nproc 65535
Step 8: Set Up a Reverse Proxy (Caddy Recommended)
Don't expose Docker ports directly to the internet. Use a reverse proxy to handle HTTPS, routing, and WAF-like protections. I prefer Caddy because it automatically provisions Let's Encrypt certificates and requires minimal configuration.
sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https
curl https://dl.cloudsmith.io/public/caddy/caddy/gpg.key | sudo gpg --dearmor -o /usr/share/keyrings/caddy-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/caddy-archive-keyring.gpg repository=https://dl.cloudsmith.io/public/caddy/caddy/deb] debian main" | sudo tee /etc/apt/sources.list.d/caddy-cloudsmith.list
sudo apt update
sudo apt install caddy -y
Create a simple Caddyfile:
sudo nano /etc/caddy/Caddyfile
Add:
nextcloud.example.com {
reverse_proxy localhost:8080 {
header_up X-Real-IP {http.request.remote.host}
header_up X-Forwarded-For {http.request.remote.host}
}
}
vaultwarden.example.com {
reverse_proxy localhost:8081
}
Replace `example.com` with your actual domain. Reload Caddy:
sudo systemctl reload caddy
sudo systemctl status caddy
Caddy automatically handles HTTPS—no manual certificate management needed.
Step 9: Enable IP Forwarding and Secure Shared Memory
For advanced networking (WireGuard, Tailscale), you might need IP forwarding. Also, secure kernel parameters to reduce attack surface:
sudo nano /etc/sysctl.conf
# Add or uncomment:
# net.ipv4.ip_forward = 1
# kernel.unprivileged_userns_clone = 0
# net.ipv4.conf.all.rp_filter = 1
# net.ipv4.icmp_echo_ignore_broadcasts = 1
# net.ipv4.tcp_syncookies = 1
# Apply changes:
sudo sysctl -p
Step 10: Backup Your Configuration
Your VPS is now hardened. Before deploying applications, create a baseline backup or snapshot. Many providers (Hetzner, Linode, DigitalOcean) offer automated snapshots. Enable them.
I also keep a simple script on my local machine to provision a fresh VPS with all these steps automated:
#!/bin/bash
# hardening.sh - Run on fresh VPS as root
set -e
echo "Updating system..."
apt update && apt upgrade -y
apt install -y ufw fail2ban curl wget git unattended-upgrades apt-listchanges
echo "Configuring firewall..."
ufw default deny incoming
ufw default allow outgoing
ufw allow 22/tcp
ufw allow 80/tcp
ufw allow 443/tcp
ufw enable
echo "Configuring fail2ban..."
systemctl enable fail2ban
systemctl start fail2ban
echo "Installing Docker..."
curl -fsSL https://