Running AI Models Locally with Ollama: Hardware Requirements and Optimization Tips

CompactHost · March 27, 2026

I spent the last three months experimenting with Ollama on my homelab, testing everything from a Raspberry Pi 4 to a used RTX 4060 Ti, and I've learned exactly what works and what doesn't. If you want to run large language models privately, without paying cloud AI fees, this is the practical guide you need. I'll walk you through real hardware specs, the gotchas I hit, and how to squeeze every bit of performance from whatever machine you're using.

Understanding Ollama and Your Hardware Reality

Ollama is a dead-simple tool for running open-source LLMs locally. You download a model, run it, and get a local API endpoint. No cloud vendor, no API keys, no monthly bill. But here's what nobody tells you upfront: the hardware you need depends heavily on which model you want to run, and some models won't work well on consumer hardware at all.

I started with Mistral-7B on a laptop with 16GB RAM and no GPU. It worked, but inference took 15–20 seconds per response. Unusable for real work. Then I added a 12GB RTX 3060, and suddenly the same model responded in 2–3 seconds. The difference is night and day, and it taught me something crucial: for practical use, you need to match your hardware to your model size and your patience threshold.

Here's the reality: Ollama can technically run on CPU alone, but you'll hate yourself. GPU acceleration is the difference between a fun experiment and something you'll actually use every day.

Minimum Hardware Requirements by Use Case

For experimentation (one-off testing): 8GB RAM, any CPU. Models like Phi-2 (2.7B) run decently. Expect 5–10 second inference times on CPU.

For daily use with smaller models: 16GB RAM minimum, and a GPU with at least 6GB VRAM. I prefer the RTX 3060 (12GB) or better. Llama 2 7B runs smoothly here at 2–3 second response times.

For running multiple concurrent models or larger 13B variants: 32GB+ RAM, and 12GB+ VRAM. I tested Mixtral 8x7B on a 24GB RTX 3090, and it's stunning—fast enough to feel conversational.

The catch: VRAM matters more than you'd think. Ollama loads the entire model into VRAM if possible, falling back to system RAM if needed. If your model doesn't fit in VRAM, it'll swap to main RAM and you'll lose 70% of your GPU benefit. I learned this the hard way trying to run a 13B model on a 6GB GPU.

GPU Acceleration: NVIDIA, AMD, and CPU Fallbacks

I've only tested Ollama seriously on NVIDIA hardware because that's what I had available, and frankly, NVIDIA's CUDA support is the most mature. Ollama detects your GPU automatically on most systems.

If you have an NVIDIA GPU with at least 2GB VRAM, Ollama will use it. The driver setup is straightforward on Linux (install nvidia-docker), less pleasant on Windows, and temperamental on macOS. I prefer running Ollama in Docker on a Linux host when possible.

AMD support via ROCm exists but requires manual setup and isn't as bulletproof. If you're buying hardware, NVIDIA is the safer bet for Ollama specifically.

For CPU-only setups, you're not out of luck—just realistic. I ran Phi-2 (2.7B) on my Ryzen 5 3600 (6-core) at about 1 token per second. Usable for summarization tasks, brutal for interactive chat. If CPU is your only option, pick a 2B or 3B model and accept slower inference.

Memory Configuration and Model Quantization

This is where most people go wrong. Running a 13B model at full precision (FP32) requires about 52GB of RAM. Your homelab doesn't have that. So we use quantization—a technique that reduces precision from 32-bit to 4-bit or 5-bit without major quality loss.

Ollama defaults to Q4 quantization, which I've found to be the sweet spot. A 13B model quantized to Q4 is about 8GB. A 7B model is about 4GB. When I switch to Q5, the quality improves slightly but file sizes jump 40%, and I rarely notice a difference in conversation quality. Q2 is too lossy for my taste; I avoid it.

Here's a Docker Compose stack I use to run Ollama with some system tuning:

version: '3.9'

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    runtime: nvidia
    environment:
      NVIDIA_VISIBLE_DEVICES: all
      OLLAMA_NUM_THREAD: 8
      OLLAMA_NUM_GPU: 1
    volumes:
      - ollama-data:/root/.ollama
    ports:
      - "11434:11434"
    devices:
      - /dev/nvidia0
      - /dev/nvidiactl
      - /dev/nvidia-uvm
    restart: unless-stopped

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    ports:
      - "8080:8080"
    environment:
      OLLAMA_BASE_URL: http://ollama:11434
    depends_on:
      - ollama
    restart: unless-stopped

volumes:
  ollama-data:

The key here: I set OLLAMA_NUM_THREAD to match your CPU core count. Ollama uses multiple threads for token generation. On my Ryzen 5 3600, I set this to 8. On your machine, count your physical cores and use that. Too high and you'll thrash the CPU scheduler; too low and you leave performance on the table.

I pair Ollama with Open WebUI (shown above) for a nice chat interface. This is my daily driver for testing models before I deploy them to production tasks.

Real Performance Benchmarks from My Setup

Let me give you actual numbers from my homelab. I tested three machines:

Machine A: Ryzen 5 3600 (6c/12t), 32GB RAM, no GPU
Mistral 7B Q4: 1.2 tokens/sec, ~40W power draw. Viable for batch jobs, miserable for chat.

Machine B: Ryzen 5 3600, 32GB RAM, RTX 3060 (12GB VRAM)
Mistral 7B Q4: 18 tokens/sec, ~120W total. Practical for daily use. Llama 2 13B Q4: 12 tokens/sec, fits entirely in VRAM.

Machine C: Ryzen 9 5950X (16c/32t), 64GB RAM, RTX 4060 Ti (16GB VRAM)
Mixtral 8x7B Q4: 22 tokens/sec. Simultaneous requests handled cleanly.

The lesson: GPU acceleration is worth every penny. The jump from Machine A to Machine B improved inference speed by 15x. CPU investment matters too, but GPU is the bottleneck for model inference in Ollama.

Optimization Tips I Use Every Day

1. Pre-load models at startup. Ollama's first request loads the model into VRAM, causing a 5–10 second delay. If you run Open WebUI behind a reverse proxy, use a health check that calls the API model endpoint. This warms up the model before users hit it.

2. Use context length wisely. By default, Ollama keeps 2048 tokens of context. Longer context = more VRAM and slower inference. For my daily chat, I reduce it to 1024 and notice no quality drop. For summarization tasks, I crank it to 4096. Test your use case.

3. Batch small requests. If you're running multiple inference jobs, batch them. Ollama handles concurrent requests, but a single batch of 10 requests runs faster than 10 sequential ones on GPU.

4. Monitor VRAM in real-time. I use nvidia-smi -l 1 in a terminal while testing. If you see your model dropping out of VRAM and swapping to RAM, your inference speed will crater. Reduce context length or quantize more aggressively.

Here's a quick script to monitor Ollama performance and log it:

#!/bin/bash
# Monitor Ollama performance every 5 seconds

while true; do
  timestamp=$(date '+%Y-%m-%d %H:%M:%S')
  vram=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits | head -1)
  cpu_percent=$(top -bn1 | grep "Cpu(s)" | sed "s/.*, *\([0-9.]*\)%* id.*/\1/" | awk '{print 100 - $1}')
  power=$(nvidia-smi --query-gpu=power.draw --format=csv,noheader,nounits | head -1)
  
  echo "$timestamp | VRAM: ${vram}MB | CPU: ${cpu_percent}% | Power: ${power}W"
  sleep 5
done

Run this in a separate terminal while testing models. It'll show you exactly when your GPU is under load and how much power you're drawing. Useful for capacity planning.

Cost Considerations and Alternatives

If you're running Ollama on existing homelab hardware, the incremental cost is near zero. But if you're building a machine specifically for this, here's what I'd spend:

Budget build ($800–$1200): Used RTX 3060 (12GB), Ryzen 5 5600X, 32GB RAM. Runs any 7B model smoothly, handles some 13B models.

Solid build ($1500–$2200): RTX 4070 Ti (12GB), Ryzen 7 5700X, 64GB RAM. Concurrent models, larger 13B models, Mixtral 8x7B comfortably.

Premium build ($3500+): RTX 4090 or 6000 Ada, threadripper CPU, 128GB RAM. Multi-model, production-grade reliability.

The alternative: rent a VPS with GPU. RackNerd offers affordable KVM VPS options including GPU-accelerated instances. I've tested their GPU offerings for brief AI workloads, and they're competitively priced if you don't need dedicated hardware. For continuous, privacy-critical workloads like mine, local Ollama wins on cost after about 2–3 months of heavy use.

Next Steps: Where to Go From Here

Once you've got Ollama running locally, the next logical step is connecting it to actual workflows. I use Open WebUI for exploration, but I also expose Ollama's REST API to Docker containers running on the same host, allowing other services to query the model programmatically.

Consider running Ollama behind a reverse proxy like Caddy or Nginx if you want to access it across your network. Just lock it down with authentication—you don't want your local LLM accessible to the internet. Pair it with Tailscale for secure remote access if you need it.

Start small: pick Mistral 7B or Llama 2 7B as your first model, run it for a week, then experiment with larger models or different quantizations. You'll quickly develop an intuition for what hardware you actually need for your specific use case.