Ollama API - Talking to Your Local Models over HTTP

Status: Active
Last Updated: 2026-08-14
Category: AImL - Phase 2: LLM Integration
Prerequisites: ollama-cli-basics
Time: 2-3 hours
Resources: [████░░░░░░] 40%
Tags: api, rest, http, generate, chat, streaming, openai-compatible, authentication

Summary

Ollama's real power is its REST API on port 11434. This lesson walks through the native /api/generate and /api/chat endpoints, streaming vs buffered responses, the OpenAI-compatible layer that lets existing tooling "just work," and how to put Ollama safely behind auth when it must be reachable beyond localhost.

---

🎯 What You'll Learn

• ✅ Call every core endpoint with curl and read the JSON responses
• ✅ Choose correctly between /api/generate (raw completion) and /api/chat (message history)
• ✅ Stream responses with Server-Sent Events and disable streaming for batch work
• ✅ Point any OpenAI-SDK application at your local server
• ✅ Expose Ollama with authentication (reverse proxy + TLS + API-key gate)

---

1. API Overview

The server listens on http://localhost:11434. Endpoints you'll actually use:

Endpoint	Purpose
POST /api/generate	Single-shot completion from a raw prompt
POST /api/chat	Multi-turn conversation from a messages array
POST /api/embeddings / POST /api/embed	Text → vector (embeddings-explained)
GET /api/tags	List downloaded models
POST /api/pull, DELETE /api/tags/...	Manage models remotely
GET /api/ps	Loaded models

Health check first:

curl http://localhost:11434/api/tags | jq '.models[].name'
# "llama3.1:8b"
# "nomic-embed-text:latest"

What Happens: No auth is required by default because Ollama binds to 127.0.0.1 only. The moment you set OLLAMA_HOST=0.0.0.0, anyone who can reach the port can run models and read model names — see §6 before exposing it.

---

2. /api/generate — Raw Completion

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1:8b",
  "prompt": "Why is the sky blue? One sentence.",
  "stream": false
}' | jq

Response (trimmed):

{
  "model": "llama3.1:8b",
  "response": "Sunlight scatters ...",
  "done": true,
  "total_duration": 2413847291,
  "eval_count": 42,
  "eval_duration": 1839471000
}

What Happens: With "stream": false the server buffers the whole generation and returns one JSON object. eval_count / eval_duration × 1e9 = tokens per second — use this for performance monitoring rather than wall-clock guesses.

Useful request fields:

Field	Meaning
"stream": true/false	SSE chunks vs single response (default true!)
"system"	System prompt without building a Modelfile
"template"	Override the prompt template (rarely needed)
"options": {...}	Any CLI parameter: temperature, num_ctx, num_predict...
"keep_alive": "30m"	Model residency after this call
"images": ["base64..."]	Vision input for multimodal models
"raw": true	Bypass templating — prompt goes in verbatim

# Options + system + no-stream in one call
curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1:8b",
  "prompt": "Classify: \"My invoice was charged twice\"",
  "system": "Reply with exactly one word: billing, bug, or question.",
  "options": {"temperature": 0, "num_predict": 5},
  "stream": false
}' | jq -r .response

---

3. /api/chat — Conversation Shape

Chat is what you want for assistants: history lives in your code, and each call sends the full message list.

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.1:8b",
  "stream": false,
  "messages": [
    {"role": "system",    "content": "You are a terse sysadmin assistant."},
    {"role": "user",      "content": "How do I see which process holds port 443?"},
    {"role": "assistant", "content": "sudo ss -tlnp | grep :443"},
    {"role": "user",      "content": "And how do I kill it?"}
  ]
}' | jq -r '.message.content'

Roles:

• system — persistent behavior instructions (first message)
• user — human turns
• assistant — previous model turns (you resend them; the model has no memory)
• tool — results fed back during tool calling (agent-frameworks)

What Happens under the hood: Ollama concatenates the messages using the model's chat template and runs one completion. Every turn re-sends full history → cost and latency grow linearly with conversation length; manage memory deliberately (context-and-tokens).

Tool-calling shape (for capable models):

{
  "model": "llama3.1:8b",
  "messages": [{"role":"user","content":"What's the disk usage on web01?"}],
  "tools": [{
    "type": "function",
    "function": {
      "name": "run_command",
      "description": "Run a shell command on a named host",
      "parameters": {
        "type": "object",
        "properties": {
          "host": {"type":"string"},
          "command": {"type":"string"}
        },
        "required": ["host","command"]
      }
    }
  }]
}

If the model decides to call a tool, .message.tool_calls comes back populated instead of (or alongside) content.

---

4. Streaming Responses

With "stream": true (the default), Ollama emits newline-delimited JSON objects — one per token chunk — then a final object with "done": true.

curl -N http://localhost:11434/api/chat -d '{
  "model": "llama3.1:8b",
  "messages": [{"role":"user","content":"Count to five."}]
}'

{"message":{"content":"One"}}
{"message":{"content":" two"}}
...
{"done_reason":"stop","done":true,"eval_count":23,...}

What Happens: This is effectively Server-Sent-Events-style chunking (Content-Type: application/x-ndjson). First-token latency ≈ model load + prompt evaluation; after that tokens arrive as generated. Streaming doesn't make generation faster overall — it makes perceived latency dramatically lower. Frontend wiring is covered in llm-streaming.

---

5. OpenAI-Compatible Endpoints

Ollama implements part of the OpenAI API surface so existing SDKs work unchanged:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")  # key ignored

resp = client.chat.completions.create(
    model="llama3.1:8b",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(resp.choices[0].message.content)

# Same thing with curl
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"llama3.1:8b",
       "messages":[{"role":"user","content":"Hi"}]}'

When to use which: the native API exposes Ollama-specific knobs (keep_alive, options.num_ctx, embeddings endpoints); the OpenAI-compatible layer buys portability — swap base_url between local Ollama and a cloud provider with zero other changes. Tools like LiteLLM build routing layers on exactly this seam.

---

6. Authentication & Safe Exposure

Native Ollama has no authentication. If it must be network-reachable, front it:

Option A — Reverse proxy with basic auth + TLS

server {
    listen 443 ssl;
    server_name ollama.internal.fogserv.cloud;

    ssl_certificate     /etc/letsencrypt/live/ollama.internal/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/ollama.internal/privkey.pem;

    auth_basic "Ollama";
    auth_basic_user_file /etc/nginx/.htpasswd;

    location / {
        proxy_pass http://127.0.0.1:11434;
        proxy_buffering off;          # REQUIRED for streaming!
        proxy_read_timeout 300s;      # long generations
    }
}

sudo htpasswd -cB /etc/nginx/.htpasswd alice   # bcrypt hashes
sudo nginx -t && sudo systemctl reload nginx

proxy_buffering off is the classic miss — with buffering on, streamed responses arrive all-at-once at the end and your UI loses its token-by-token feel.

Option B — WireGuard-only access

Keep Ollama bound to LAN/tailnet interfaces only; no public exposure at all. See security/wireguard-vpn. For homelabs this is usually the right answer.

Additional hardening regardless of option:

• Firewall: allow only trusted sources to 11434 (security/firewall-basics)
• Put generation stats behind monitoring ([observability/prometheus-introduction] if built) — eval_count/eval_duration make great metrics
• Remember: whoever can query the model can also exfiltrate anything your RAG index contains

---

🛠️ Troubleshooting & Common Issues

Symptom	Cause	Fix
curl: (7) connection refused	Server down or wrong host/port	systemctl status ollama; check OLLAMA_HOST
Works locally, refused from another box	Bound to 127.0.0.1	Set OLLAMA_HOST=0.0.0.0, restart, firewall appropriately
Response arrives only at the end	Proxy buffering or client reading whole body	proxy_buffering off; consume NDJSON incrementally
First request slow (~seconds), rest fast	Cold model load	Warm up at deploy time; tune keep-alive
500 with template errors after custom "template"	Malformed Go template	Remove override; use default unless you know the format
OpenAI SDK says model not found	Model name mismatch	Names must match ollama list exactly (llama3.1:8b, not gpt-4)
Truncated answers via SDK	Client max_tokens default low	Set max_tokens explicitly
Random 401s through proxy	Multiple auth backends / stale htpasswd	Check nginx error log; regenerate file

---

🔗 Related

• Prev: ollama-cli-basics
• Next: python-llm-integration — proper client code
• Also: llm-streaming, context-and-tokens
• Infra: [networking/traefik-setup], security/tls-configuration, security/wireguard-vpn

---

📝 Change Log

• 2026-08-14 — Initial publication. Covers README outline: REST overview, /api/generate, /api/chat, streaming, API authentication.