Getting Started with Ollama: Running Large Language Models Locally with Docker

Getting Started with Ollama: Running Large Language Models Locally with Docker

CompactHost ·

We earn commissions when you shop through the links on this page, at no additional cost to you. Learn more.

Running a large language model on your own hardware felt like science fiction three years ago. Today, with Ollama, you can have llama3.2 or mistral answering questions from your terminal in under ten minutes — no cloud account, no API key, no usage bill. I run Ollama on a mini PC with 32 GB of RAM and it handles day-to-day coding questions and document summarisation well enough that I rarely reach for a cloud API anymore.

In this tutorial I'll walk you through getting Ollama running inside Docker, pulling a few models, wiring up a GPU if you have one, and then adding Open WebUI so you get a proper chat interface instead of a bare REST endpoint. I'll keep things focused on Docker because it makes the whole stack reproducible and easy to update.

Why Docker Instead of the Native Installer?

Ollama ships a native installer for Linux, macOS, and Windows, and it works fine. I still prefer Docker for homelab use because I can version-pin the image, restart it alongside my other compose stacks, and tear it down without leaving anything behind on the host. It also makes GPU passthrough explicit and auditable — you know exactly what the container is touching.

The one real trade-off is a slightly more involved GPU setup. I'll cover both CPU-only and NVIDIA GPU configurations below.

Prerequisites

Tip: If you're on a CPU-only machine, start with a 3B parameter model like llama3.2:3b or qwen2.5:3b. They're surprisingly capable and will actually respond at a usable speed — typically 5–15 tokens per second on a modern x86 CPU.

Step 1: Create the Docker Compose File

I keep all my Ollama config in ~/stacks/ollama/. Create that directory and drop in the following compose.yml. The CPU-only version is the simplest starting point:

mkdir -p ~/stacks/ollama
cd ~/stacks/ollama
nano compose.yml
services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama
    environment:
      - OLLAMA_KEEP_ALIVE=24h
      - OLLAMA_NUM_PARALLEL=2

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    restart: unless-stopped
    ports:
      - "3000:8080"
    volumes:
      - open_webui_data:/app/backend/data
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    depends_on:
      - ollama

volumes:
  ollama_data:
  open_webui_data:

A few things I want to call out here. OLLAMA_KEEP_ALIVE=24h tells Ollama to keep a loaded model in memory for 24 hours instead of unloading it after a few minutes of inactivity — this is the single biggest quality-of-life improvement for interactive use. OLLAMA_NUM_PARALLEL=2 allows two simultaneous inference requests, which is useful once Open WebUI is in the picture. The models directory is stored in a named Docker volume at /root/.ollama inside the container, so your downloaded models survive container recreations.

Step 2: Add GPU Passthrough (NVIDIA)

If you have an NVIDIA card, first make sure the container toolkit is installed on the host:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
  sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt update && sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Then add the deploy section to the ollama service in your compose file:

  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama
    environment:
      - OLLAMA_KEEP_ALIVE=24h
      - OLLAMA_NUM_PARALLEL=2
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
Watch out: The deploy.resources GPU passthrough syntax only works with docker compose (v2). If you're running the old docker-compose (v1) Python binary, you'll need the legacy runtime: nvidia key instead. Check your version with docker compose version.

Step 3: Start the Stack and Pull a Model

Bring everything up:

docker compose up -d
docker compose logs -f ollama

Watch the logs for a moment to confirm Ollama started cleanly — you should see something like Listening on [::]:11434. Then pull your first model. I always start with llama3.2:3b to confirm things are working before committing to a larger download:

# Pull a 3B model (~2 GB download)
docker exec -it ollama ollama pull llama3.2:3b

# Try a quick inference from the command line
docker exec -it ollama ollama run llama3.2:3b "What is self-hosting?"

# Pull a more capable 7B model when you're ready
docker exec -it ollama ollama pull mistral:7b

# List everything you have downloaded
docker exec -it ollama ollama list

The first response from ollama run will take a few seconds while the model loads into memory. Subsequent requests in the same session are much faster because the model stays loaded thanks to the OLLAMA_KEEP_ALIVE variable we set earlier.

Step 4: Use the Open WebUI Interface

Open a browser and navigate to http://your-server-ip:3000. On first launch, Open WebUI will ask you to create an admin account — this is purely local, nothing leaves your machine. After logging in, click the model selector at the top of the chat window and you should see every model you pulled through Ollama listed there automatically.

Open WebUI has come a long way. I use it daily for document Q&A (it supports uploading PDFs and text files directly into the chat), quick code reviews, and drafting emails. The RAG pipeline built into recent versions is genuinely useful — you can point it at a folder of markdown notes and ask questions against the whole collection.

Step 5: Expose Ollama Securely (Optional)

By default, port 11434 only accepts connections from localhost inside Docker's bridge network. The compose file above publishes it on all interfaces (0.0.0.0:11434), which is fine on a private LAN but you should not expose this port directly to the internet — Ollama has no built-in authentication.

If you want to access your Ollama instance remotely, I recommend one of two approaches: tunnel it over Tailscale (zero config, zero exposed ports) or put it behind Caddy or Nginx Proxy Manager with HTTP basic auth or Authelia in front. I personally use Tailscale for Ollama — the latency is low enough that streaming token output still feels snappy over a Tailnet connection.

Useful Model Recommendations

Once the stack is running, here are the models I actually keep around and why:

# Pull the embedding model for Open WebUI RAG
docker exec -it ollama ollama pull nomic-embed-text

# Remove a model you no longer want (frees disk space immediately)
docker exec -it ollama ollama rm codellama:7b

Keeping Everything Updated

I use Watchtower to keep my images current, but Ollama image updates are worth reviewing manually because they sometimes change default behaviour. A safer pattern is a quick manual pull every few weeks:

cd ~/stacks/ollama
docker compose pull
docker compose up -d

Note that pulling a new Ollama image does not update the model weights themselves — those live in the ollama_data volume and are managed separately with ollama pull. You'll need to re-pull models that have received updates (like Llama or Mistral releases) manually.

Tip: The Ollama model library at ollama.com/library lists every available model with size, license, and capability notes. Check it whenever a new model release makes the news — popular models like Llama and Gemma typically appear within days of their official launch.

Wrapping Up

You now have a fully self-hosted LLM stack: Ollama serving models over a local REST API, and Open WebUI giving you a polished chat interface that your whole household or team can use. The entire thing runs in two containers, persists its data in named volumes, and can be updated with a single docker compose pull && docker compose up -d.

From here, the natural next steps are adding a reverse proxy with SSL so you can reach it from anywhere on your Tailnet with a proper domain name, and exploring the Ollama REST API directly — it's OpenAI-compatible, which means most tools written for the OpenAI SDK will work against your local stack with a one-line URL change.

Discussion