fogserv.cloud fogserv.cloud

Ollama Setup - Run LLMs Locally in Minutes

Status: Active
Last Updated: 2026-08-14
Category: AImL - Phase 1: LLM Fundamentals
Prerequisites: llm-introduction, containers/docker-basics
Time: 1-2 hours
Resources: 8GB+ RAM, GPU optional
Tags: ollama, installation, gpu, cpu, inference, model-management, self-hosted

Summary

Install Ollama (v0.32+) — the simplest way to run open-weight LLMs on your own hardware — verify whether you're on CPU or GPU inference, download your first models, and learn how Ollama stores and manages them on disk. Covers native Linux install, Docker deployment, GPU detection, model management, and new v0.32+ features like model metadata caching and improved onboarding.

🎯 What You'll Learn

1. What Ollama Actually Is

Ollama bundles four things into one binary:

Component Job
Model runner Loads GGUF weights into RAM/VRAM (built on llama.cpp)
Inference engine Token generation, KV-cache management, batching
API server REST API on port 11434 (OpenAI-compatible endpoints included)
Model registry client ollama pull fetches from ollama.com like Docker pulls images

If you know Docker, the mental model is nearly identical:

docker:   daemon → containers ← images → registries
ollama:   server → loaded models ← downloaded weights → ollama.com

2. Installation

2.1 Linux (Native) — Recommended for Servers

curl -fsSL https://ollama.com/install.sh | sh

What Happens: The script detects your distro and GPU, downloads the release tarball (v0.32+ as of 2025), installs to /usr/local, creates an ollama systemd service (running as the ollama user), and enables it at boot. v0.32+ adds improved model metadata caching between requests, cutting first-load latency on repeat pulls.

Verify:

systemctl status ollama
# ● ollama.service - Ollama Service
#      Loaded: loaded (/etc/systemd/system/ollama.service; enabled)

ollama --version
# ollama version is 0.32.x

2.2 Manual Install (No curl-pipe-sh)

curl -LO https://ollama.com/download/ollama-linux-amd64.tgz
sudo tar -xzf ollama-linux-amd64.tgz -C /usr
sudo useradd -r -s /bin/false -m -d /usr/share/ollama ollama
sudo tee /etc/systemd/system/ollama.service > /dev/null <<'EOF'
[Unit]
Description=Ollama Service
After=network-online.target

[Service]
ExecStart=/usr/bin/ollama serve
User=ollama
Group=ollama
Restart=always
RestartSec=3
Environment="PATH=/usr/bin:/usr/local/bin"
Environment="OLLAMA_HOST=0.0.0.0"

[Install]
WantedBy=default.target
EOF
sudo systemctl daemon-reload && sudo systemctl enable --now ollama

OLLAMA_HOST=0.0.0.0 exposes the API beyond localhost — only do this behind a firewall/reverse proxy (security/network-segmentation). The default binds to 127.0.0.1 only.

2.3 Docker Alternative

# CPU-only
docker run -d --name ollama -p 11434:11434 \
  -v ollama:/root/.ollama ollama/ollama:latest

# NVIDIA GPU
docker run -d --name ollama -p 11434:11434 \
  --gpus=all \
  -v ollama:/root/.ollama ollama/ollama:latest

# Quick test inside the container
docker exec -it ollama ollama run llama3.2:3b "hello"

When to prefer Docker: you already run everything containerized (see containers/docker-basics) and want consistent deploys via compose. When native: bare-metal GPU boxes where you want the least overhead between driver and model.

Note: Use ollama/ollama:latest or pin to a specific version like ollama/ollama:0.32. The Docker image is updated frequently; check Docker Hub for the latest.

2.4 macOS / Windows

Download the app from https://ollama.com/download. It runs in the menu bar/system tray and serves the same 127.0.0.1:11434 API. Fine for development; this course focuses on Linux servers.

3. GPU vs CPU Inference

Ollama auto-detects hardware at startup. Check the logs:

journalctl -u ollama -e | grep -iE 'gpu|cuda|rocm|vram'

GPU output looks like:

ggml_cuda_init: found 1 CUDA devices
looking for compatible GPUs ...
using CUDA0 (NVIDIA GeForce RTX 3090) ... 24576 MiB VRAM

CPU-only shows no device lines — inference still works, just slower.

Speed Reality Check (7B Q4 model, tokens/sec)

Hardware Prompt eval Generation Feel
RTX 4090 / 3090 thousands t/s ~60–100 t/s Instant
RTX 3060 12GB fast ~35–45 t/s Very usable
M-series Mac (unified mem) varies ~20–40 t/s Usable
Ryzen desktop CPU (DDR5) slow ~8–15 t/s Tolerable for chat
Older laptop CPU very slow ~2–5 t/s Painful

Rule of thumb: below ~7–10 tokens/sec, interactive use feels broken.

GPU Memory Sizing

A model must fit in VRAM to run fully on GPU. Q4 quantized sizes:

Model Size Fits in
llama3.2:3b ~2 GB Any 4GB+ card
llama3.1:8b ~4.9 GB 6–8GB cards
qwen2.5:14b ~9 GB 12GB cards
gemma2:27b / qwen2.5:32b ~17–20 GB 24GB cards

Partial offload happens automatically when the model doesn't fit (some layers on GPU, rest on CPU) — functional but much slower. Monitor with nvidia-smi.

v0.32+ Note: Improved GPU memory management with better flash attention support. Set OLLAMA_FLASH_ATTENTION=1 for VRAM savings on supported GPUs (Ampere+). See Ollama GPU docs for current GPU support matrix including AMD ROCm and Apple Metal.

4. Downloading Models and First Inference

# Pull a small model first to validate the install
ollama pull llama3.2:3b

# Run it interactively
ollama run llama3.2:3b
>>> Explain VLANs to me in one paragraph.

What Happens: run auto-pulls if missing, loads weights into memory (watch the progress bar on first load — subsequent loads are near-instant thanks to caching), then streams a response token by token.

Then try a bigger one if you have the RAM:

ollama pull llama3.1:8b     # the course workhorse (~4.9 GB)
ollama run llama3.1:8b
>>> /bye                     # exits the session

Useful flags during a chat session:

Command Effect
/show info Model metadata (params, quantization, context length)
/set parameter temperature 0.3 Tune sampling for this session
/set system "You are a terse sysadmin assistant" Set system prompt for session
/clear Reset conversation context
/bye Exit

Non-interactive one-shot:

ollama run llama3.2:3b "Summarize: $(cat notes.txt)"

5. Model Management

ollama list        # downloaded models + sizes + IDs
ollama ps          # which models are currently LOADED in memory
ollama cp llama3.1:8b my-assistant:v1       # tag a copy
ollama rm llama3.2:3b                       # delete
ollama pull llama3.1:8b                      # update to latest digest

What Happens with ollama ps: Ollama keeps recently-used models in memory (default keep-alive 5 minutes). A loaded model answers instantly; an unloaded one pays a load penalty of seconds-to-tens-of-seconds. Control it:

# Keep the model resident for 2 hours after last use
OLLAMA_KEEP_ALIVE=2h ollama run llama3.1:8b
# Or per request via API: {"model": "...", "keep_alive": "2h"}

Where Models Live on Disk

Install type Path
Native (service) /usr/share/ollama/.ollama/models
Native (your user) ~/.ollama/models
Docker inside container /root/.ollama/models (mount a volume!)

Layout: blobs/ holds content-addressed weight files (the big ones); manifests/ holds the tag→blob mappings. Disk fills up fast — a few 30B-class experiments can eat 100GB+. Check with du -sh ~/.ollama (or the service path) and prune with ollama rm.

v0.32+: Model metadata is now cached between requests, reducing startup latency for repeated model loads. The cache lives in the same model directory.

6. Service Configuration Cheat Sheet

Set via systemd override (sudo systemctl edit ollama) under [Service]:

Variable Purpose Example
OLLAMA_HOST Bind address:port 0.0.0.0:11434
OLLAMA_MODELS Move model storage (e.g., to a big disk) /data/ollama/models
OLLAMA_KEEP_ALIVE Default residency 30m
OLLAMA_MAX_LOADED_MODELS Concurrent models in memory 2
OLLAMA_NUM_PARALLEL Parallel requests per model 4
OLLAMA_FLASH_ATTENTION Enable flash attention (VRAM savings) 1
OLLAMA_KV_CACHE_TYPE KV cache quantization q8_0

After editing: sudo systemctl restart ollama.

Moving model storage example (common on homelabs where / is small):

Environment="OLLAMA_MODELS=/mnt/bigdisk/ollama"

Remember to chown -R ollama:ollama /mnt/bigdisk/ollama so the service can write there.

🛠️ Troubleshooting & Common Issues

Symptom Likely cause Fix
ollama: command not found after install script PATH lacks /usr/local/bin Re-login or check install log for errors
Service starts but connection refused :11434 Service not running / different host systemctl status ollama; are you on the right machine?
GPU detected in nvidia-smi but not by Ollama Missing runtime libs Re-run installer (it installs CUDA deps); check journalctl -u ollama
Extremely slow first answer, then fine Cold load from disk Increase OLLAMA_KEEP_ALIVE; put models on SSD/NVMe
Error: out of memory on load Model + KV cache > VRAM/RAM Smaller model or lower quantization; close other loaded models (ollama ps)
Docker container loses all models on recreate No persistent volume Mount -v ollama:/root/.ollama
Downloads fail mid-pull Registry hiccup Just re-run ollama pull — resumes from partial blobs
API unreachable from other machines Binds to 127.0.0.1 Set OLLAMA_HOST=0.0.0.0, then firewall appropriately

🔗 Related

📝 Change Log

Choose Theme

Your selection is saved locally.

Neural Cacophony
Aperture v2
Flux v1
Mosaic Chaos
Nexus v1
Nexus Zest
Prism v2
Synapse