Setting Up Ollama with Docker Compose for Local LLM Inference on a VPS

CompactHost · July 19, 2026

We earn commissions when you shop through the links on this page, at no additional cost to you. Learn more.

Running a local LLM on a VPS gives you something cloud AI APIs simply can't: full data privacy, no per-token billing, and the freedom to experiment with models at 3 AM without a credit card bill waiting for you in the morning. Ollama makes model management surprisingly painless, and wrapping it in Docker Compose means your entire inference stack lives in a single version-controlled file. In this tutorial I'll walk you through a production-ready Docker Compose setup for Ollama on a VPS — including persistent model storage, a Caddy reverse proxy with automatic HTTPS, and a few hard-won lessons about memory limits and model preloading.

Choosing the Right VPS for Ollama

Before we touch a single config file, let me be honest about hardware requirements. Ollama runs models entirely in CPU-mode on most VPS instances, since GPU Droplets and dedicated GPU VMs are a different price tier. For CPU inference you need RAM above all else — a 7B parameter model in Q4 quantisation needs roughly 5–6 GB of RAM free, and a 13B model needs closer to 10 GB. That means a 4 GB VPS will struggle with anything beyond the smallest models like Phi-3 mini or Gemma 2B.

I personally run Ollama on a DigitalOcean Droplet with 8 vCPUs and 16 GB of RAM, which comfortably handles Llama 3.1 8B and Mistral 7B simultaneously. DigitalOcean's Droplets come with predictable monthly pricing and a 99.99% SLA — important when you're running an inference endpoint that other services depend on. If you're just getting started and want to spin up a cloud server without the complexity, create your DigitalOcean account here and look at the General Purpose 8 GB or 16 GB plans.

Project Structure

I like keeping everything under /opt/ollama on the server. It keeps permissions simple and makes backups obvious. SSH in and create the directory layout:

sudo mkdir -p /opt/ollama/{data,caddydata,caddyconfig}
sudo chown -R $USER:$USER /opt/ollama
cd /opt/ollama

We'll store all downloaded models under /opt/ollama/data, which gets bind-mounted into the container. This means models survive container restarts and image updates — something that trips up a lot of people who use anonymous Docker volumes and then wonder where their 4 GB model download went.

The Docker Compose File

Here's the full docker-compose.yml I use. It runs Ollama alongside Caddy as the reverse proxy. I prefer Caddy because it handles Let's Encrypt certificates automatically with zero configuration — no certbot cron jobs, no renewal hooks, just declare your domain and move on.

cat > /opt/ollama/docker-compose.yml <<'EOF'
services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    volumes:
      - ./data:/root/.ollama
    environment:
      - OLLAMA_HOST=0.0.0.0
      - OLLAMA_NUM_PARALLEL=2
      - OLLAMA_MAX_LOADED_MODELS=2
    ports:
      - "127.0.0.1:11434:11434"
    deploy:
      resources:
        limits:
          memory: 12G

  caddy:
    image: caddy:2-alpine
    container_name: caddy
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - ./caddydata:/data
      - ./caddyconfig:/config
    depends_on:
      - ollama

networks:
  default:
    name: ollama_net
EOF

A few things worth calling out here. I bind Ollama to 127.0.0.1:11434 rather than 0.0.0.0:11434 — this means the API port is never directly exposed to the internet, only to Caddy inside the Docker network and to the host loopback interface. The OLLAMA_NUM_PARALLEL=2 variable lets two inference requests run concurrently, which is useful if you're serving multiple users or running automated pipelines. The memory limit of 12G gives the container breathing room on a 16 GB host while preventing it from consuming everything if a runaway request tries to load too large a model.

Configuring the Caddyfile

Replace llm.yourdomain.com with your actual subdomain. Make sure you've pointed an A record at your VPS IP before running Compose, otherwise Caddy's ACME challenge will fail.

cat > /opt/ollama/Caddyfile <<'EOF'
llm.yourdomain.com {
    reverse_proxy ollama:11434

    basicauth /* {
        # Generate hash with: caddy hash-password --plaintext yourpassword
        yourusername JDJhJDE0JEFCQzEyM...hashgoeshere
    }

    log {
        output file /data/access.log
        format json
    }
}
EOF

The basicauth block adds HTTP Basic Authentication in front of the Ollama API. Ollama itself has no built-in authentication, so without this block anyone who finds your domain can run inference on your server and rack up CPU usage. Generate the password hash with docker run --rm caddy:2-alpine caddy hash-password --plaintext 'yourpassword' and paste it in.

Starting the Stack and Pulling Your First Model

Bring everything up with:

cd /opt/ollama
docker compose up -d

# Watch the logs to confirm Caddy gets its certificate
docker compose logs -f caddy

# Once healthy, pull a model — this runs inside the container
docker exec -it ollama ollama pull llama3.1:8b

# Verify the model loaded correctly
docker exec -it ollama ollama list

The first ollama pull will download several gigabytes. On a VPS with a fast uplink (DigitalOcean Droplets typically see 1–5 Gbps to their registry mirrors) this usually takes under two minutes for a 7B Q4 model. On a slower provider it can take considerably longer — plan accordingly.

Once the model is downloaded, test your authenticated endpoint from your local machine:

# Replace with your actual domain and credentials
curl -u yourusername:yourpassword \
  https://llm.yourdomain.com/api/generate \
  -d '{
    "model": "llama3.1:8b",
    "prompt": "Explain Docker networking in two sentences.",
    "stream": false
  }'

You should get a JSON response with the model's output within a few seconds. CPU inference on a 8-vCPU Droplet typically produces 10–20 tokens per second for a 7B model — fast enough for interactive use, though not as snappy as a GPU.

Keeping Models Warm with a Preload Script

By default Ollama unloads models from memory after five minutes of inactivity. That means the first request after a cold period incurs a multi-second load delay. I solve this with a simple preload that runs after the container starts:

cat > /opt/ollama/preload.sh <<'EOF'
#!/bin/bash
# Ping the model to keep it loaded — run this at container startup
sleep 10  # Give Ollama time to initialise
curl -s http://localhost:11434/api/generate \
  -d '{"model":"llama3.1:8b","prompt":"ping","stream":false,"keep_alive":"1h"}' \
  > /dev/null
echo "Model preloaded and kept alive for 1 hour"
EOF
chmod +x /opt/ollama/preload.sh

# Add to crontab to run on reboot
(crontab -l 2>/dev/null; echo "@reboot sleep 30 && /opt/ollama/preload.sh") | crontab -

The keep_alive parameter in the API request tells Ollama to keep the model resident in memory for the specified duration. Setting it to -1 keeps it loaded indefinitely, which is fine if this is your only model and you have the RAM for it.

Updating Ollama Automatically with Watchtower

Ollama releases updates fairly frequently — new model format support, performance improvements, bug fixes. Rather than SSH-ing in every week to pull the latest image, I add Watchtower to the stack. I configure it to update only the Ollama container automatically and leave Caddy on manual updates (Caddy changes can occasionally affect Caddyfile syntax).

Add this service to your docker-compose.yml under the existing services:

  watchtower:
    image: containrrr/watchtower:latest
    container_name: watchtower
    restart: unless-stopped
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - WATCHTOWER_POLL_INTERVAL=86400
      - WATCHTOWER_CLEANUP=true
      - WATCHTOWER_INCLUDE_STOPPED=false
    command: ollama

The trailing ollama argument tells Watchtower to only monitor that specific container. It checks for updates once every 24 hours (86400 seconds) and cleans up old images automatically so your VPS disk doesn't fill up with stale layers.

Monitoring RAM and Disk Usage

Ollama's biggest operational risk on a VPS is disk exhaustion — large models accumulate fast when you're experimenting. I run a quick check in my weekly routine:

# See how much space models are consuming
docker exec ollama ollama list

# Disk usage of the model directory
du -sh /opt/ollama/data/models/

# Current memory pressure
docker stats ollama --no-stream

# Remove a model you no longer need
docker exec ollama ollama rm mistral:7b

A good rule of thumb: keep 20% of your VPS disk free at all times. If you're on a 50 GB plan and have pulled three 4–5 GB models, you're already at around 30% utilised — add system packages, Docker images, and logs and you can hit the wall faster than you expect.

Next Steps

With Ollama running behind Caddy and serving authenticated HTTPS requests, you've got a solid foundation. From here, the natural progression is adding Open WebUI as a chat frontend — it speaks directly to the Ollama API and gives you a familiar ChatGPT-style interface for testing models. You might also look at connecting this Ollama endpoint to n8n or Home Assistant for automation workflows, or wiring it into VS Code's Continue extension as a private code completion backend.

The stack we've built here is genuinely production-capable. I've been running a nearly identical setup on DigitalOcean for months without a restart that wasn't voluntary. If you're not yet on a cloud host that can handle this workload, DigitalOcean lets you build and deploy from code to production in just a few clicks — their General Purpose Droplets with 16 GB RAM are my go-to for CPU inference workloads.