Python LLM Integration - Calling Ollama from Python

Status: Active
Last Updated: 2026-08-26
Category: AI/ML - Phase 2: Local LLMs
Prerequisites: ollama-api, ollama-setup
Time: 2 hours
Tags: python, ollama, api, streaming, json-mode, openai-compatible

Summary

Wire your Python applications directly into locally-hosted models on Ollama. This article covers the two main client paths (the OpenAI-compatible endpoint and the official ollama Python library), streaming responses, structured JSON output, and production hardening with retries and timeouts.

🎯 What You'll Learn

By the end of this article, you'll be able to:

✅ Choose between the OpenAI-compatible client and the ollama library
✅ Stream completions token-by-token in Python
✅ Force structured JSON output with format: json and JSON Schema
✅ Add timeouts, retries, and connection-error handling that survive Ollama restarts
✅ Build a thin wrapper module you can reuse across homelab projects

Context / Why This Matters

ollama-api showed the raw HTTP surface of Ollama. Real applications need more than curl: typed clients, streaming UIs, guaranteed-parseable output for downstream code, and resilience when the model server is busy loading a model or restarting.

Everything here runs against your own hardware — no API keys, no egress, no per-token billing. That makes local Ollama the right default for homelab automation, log triage, and internal tools where sending data to a cloud vendor is a non-starter. If you later need a hosted model, the OpenAI-compatible path means swapping one base URL and one key.

Implementation / Core Content

Two Client Paths

	`openai` client (OpenAI-compat endpoint)	`ollama` library
Base URL	`http://localhost:11434/v1`	`http://localhost:11434`
Auth	Any dummy API key (`ollama`)	None
Streaming	Yes (SSE chunks)	Yes
Structured output	`response_format={"type": "json_object"}` / json_schema	`format="json"` or `format=<pydantic schema>`
Model listing/embeddings	Yes	Yes (`list()`, `embeddings()`)
Portability to cloud	Excellent — change base_url only	Locked to Ollama

Install either:

pip install openai          # OpenAI-compatible path
pip install ollama          # Official Ollama client

Path 1: OpenAI-Compatible Client

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama",  # required but ignored by Ollama
)

resp = client.chat.completions.create(
    model="llama3.1:8b",
    messages=[
        {"role": "system", "content": "You are a concise homelab assistant."},
        {"role": "user", "content": "Explain what a reverse proxy does in 3 sentences."},
    ],
    temperature=0.7,
    max_tokens=300,
)
print(resp.choices[0].message.content)

The big advantage: moving to a hosted provider later is changing base_url and api_key. All application code stays identical.

Path 2: The `ollama` Library

import ollama

resp = ollama.chat(
    model="llama3.1:8b",
    messages=[
        {"role": "system", "content": "You are a concise homelab assistant."},
        {"role": "user", "content": "Explain what a reverse proxy does in 3 sentences."},
    ],
)
print(resp["message"]["content"])

The native client exposes Ollama-specific features cleanly: keep_alive (how long the model stays loaded), options (num_ctx, temperature, seed), and native Pydantic-based structured output.

Streaming

Streaming matters for perceived latency — first token typically arrives in well under a second on a warm model, versus many seconds for a full completion.

# OpenAI-compatible streaming
stream = client.chat.completions.create(
    model="llama3.1:8b",
    messages=[{"role": "user", "content": "Summarize last night's backup job logs."}],
    stream=True,
)
for chunk in stream:
    delta = chunk.choices[0].delta.content or ""
    print(delta, end="", flush=True)
print()

# Native client streaming
import ollama

stream = ollama.chat(
    model="llama3.1:8b",
    messages=[{"role": "user", "content": "Summarize last night's backup job logs."}],
    stream=True,
)
for part in stream:
    print(part["message"]["content"], end="", flush=True)

Accumulate chunks into one string if you also need the full text afterward (logging, post-processing):

pieces = []
for part in stream:
    pieces.append(part["message"]["content"])
full_text = "".join(pieces)

Structured Output (JSON Mode)

Freeform LLM text is fine for humans but hostile to code. Two escalation levels:

Level 1 — format="json": guarantees valid JSON, not any particular shape.

import json
import ollama

resp = ollama.chat(
    model="llama3.1:8b",
    messages=[
        {"role": "system", "content": 'Extract incidents as JSON: {"incidents": [{"service": str, "severity": "low|medium|high"}]}'},
        {"role": "user", "content": log_excerpt},
    ],
    format="json",
)
data = json.loads(resp["message"]["content"])

Always describe the desired schema in the prompt even with format="json" — otherwise you get valid JSON of arbitrary shape.

Level 2 — JSON Schema via Pydantic: guarantees the exact shape (Ollama ≥0.5 constrained decoding):

from pydantic import BaseModel
from typing import Literal
import ollama

class Incident(BaseModel):
    service: str
    severity: Literal["low", "medium", "high"]
    action: str

class IncidentReport(BaseModel):
    incidents: list[Incident]

resp = ollama.chat(
    model="llama3.1:8b",
    messages=[{"role": "user", "content": log_excerpt}],
    format=IncidentReport.model_json_schema(),
)
report = IncidentReport.model_validate_json(resp["message"]["content"])

On the OpenAI-compatible endpoint, use response_format={"type": "json_object"} plus an explicit schema in the system prompt, then validate with Pydantic yourself. Validation failures should be retried (below), never silently swallowed.

Timeouts, Retries, and Resilience

Local inference has failure modes cloud SDKs hide from you:

Model load stall — first request after idle pulls gigabytes from disk; can take 30–60s+ on spinning rust.
Connection refused — Ollama service restarted or still booting.
Context overflow — prompt exceeds num_ctx; request errors instead of truncating.
Slow generation — large prompts on CPU-only hosts can take minutes.

import time
import httpx
import ollama
from pydantic import ValidationError

def chat_with_retry(messages, *, model="llama3.1:8b", max_retries=3,
                    timeout=120.0, retryable=(httpx.ConnectError, httpx.ReadTimeout)):
    last_exc = None
    for attempt in range(1, max_retries + 1):
        try:
            client = ollama.Client(host="http://localhost:11434", timeout=timeout)
            return client.chat(model=model, messages=messages, keep_alive="10m")
        except retryable as exc:
            last_exc = exc
            backoff = min(2 ** attempt, 15)
            print(f"[attempt {attempt}] {exc!r}; retrying in {backoff}s")
            time.sleep(backoff)
    raise RuntimeError(f"Ollama unreachable after {max_retries} attempts") from last_exc

def chat_validated(messages, schema_model, *, attempts=3, **kw):
    """Ask for JSON, validate, re-prompt once with the error if malformed."""
    msgs = list(messages)
    for _ in range(attempts):
        resp = chat_with_retry(msgs, **kw)
        try:
            return schema_model.model_validate_json(resp["message"]["content"])
        except ValidationError as exc:
            msgs = msgs + [
                {"role": "assistant", "content": resp["message"]["content"]},
                {"role": "user", "content": f"That was invalid: {exc}. "
                                            "Return ONLY corrected JSON matching the schema."},
            ]
    raise RuntimeError("Model failed to produce valid output")

Key settings:

Timeout ≥ 120s for cold starts; keep it shorter (30s) for interactive paths where a warm model is guaranteed.
Keep keep_alive="10m" (or longer) on services that call the model frequently, so you don't pay the load cost every call.
Retry only idempotent calls, and cap total latency budget, not just attempt count.

Practical Examples

Example 1: Homelab Log Triage Script

Reads journalctl output, returns structured incidents:

#!/usr/bin/env python3
"""triage.py — summarize failed units from journalctl."""
import json, subprocess, ollama
from pydantic import BaseModel

class Report(BaseModel):
    summary: str
    failing_units: list[str]

logs = subprocess.run(
    ["journalctl", "-p", "err", "-b", "--no-pager", "-n", "100"],
    capture_output=True, text=True,
).stdout[-6000:]  # stay well under default context

resp = ollama.chat(
    model="llama3.1:8b",
    messages=[
        {"role": "system", "content": "You are a Linux sysadmin assistant. Reply in JSON."},
        {"role": "user", "content": f"Summarize these error logs:\n\n{logs}"},
    ],
    format=Report.model_json_schema(),
    options={"num_ctx": 8192, "temperature": 0.2},
)
report = Report.model_validate_json(resp["message"]["content"])
print(report.summary)
for unit in report.failing_units:
    print(f" - {unit}")

Expected behavior: prints a 1–2 sentence summary plus unit names like postgres.service, backup.timer.

Example 2: Streaming CLI Chat

import ollama

history = [{"role": "system", "content": "You are a helpful homelab assistant."}]
while True:
    try:
        user = input("\nyou> ").strip()
    except (EOFError, KeyboardInterrupt):
        break
    if not user:
        continue
    history.append({"role": "user", "content": user})
    reply = ""
    for part in ollama.chat(model="llama3.1:8b", messages=history, stream=True):
        piece = part["message"]["content"]
        reply += piece
        print(piece, end="", flush=True)
    history.append({"role": "assistant", "content": reply})

Note history grows unboundedly — see context-and-tokens for trimming strategies.

Example 3: Batch Embeddings for Later Retrieval

import ollama

texts = [chunk_text for chunk_text in my_document_chunks]
vectors = ollama.embed(model="nomic-embed-text", input=texts)["embeddings"]
# Feed vectors + chunk ids into Qdrant — see qdrant-setup.md

Troubleshooting & Common Pitfalls

Problem	Cause	Fix
`Connection refused` to localhost:11434	Ollama not running, or app runs in a container (localhost ≠ host)	`systemctl status ollama`; from containers use `host.docker.internal` or the host IP
First request takes ~60s	Cold model load from disk	Pre-warm at boot (`ollama run model --keepalive` or a health-check ping); set generous timeouts
`json.loads` fails despite `format="json"`	Response valid JSON but wrong shape; or older Ollama without support	Use JSON Schema/Pydantic format; upgrade Ollama; add validate-and-retry loop
Output truncated mid-sentence	Hit `num_predict` (default limit) or context window	Raise `options.num_predict`; check prompt size vs `num_ctx` (context-and-tokens)
404 model not found	Model tag typo, or model not pulled	`ollama list`; use exact tags like `llama3.1:8b`, not `llama3.1` blindly
Streaming loop yields nothing until end	Proxy buffering SSE	Connect directly to Ollama's port; disable proxy buffering for `/v1/` paths
Requests serialize slowly under concurrency	One GPU, models queue	Accept queuing, or run a smaller quant — see model-quantization

Next Steps / Ops Actions

Structure prompts deliberately before scaling usage: prompt-engineering
Understand exactly how much context those scripts consume: context-and-tokens
Persist extracted knowledge into a vector store: qdrant-setup
Wrap the Ollama container alongside your app with compose: docker-compose-intro

Sources & Related

External references consulted:

https://github.com/ollama/ollama/blob/main/docs/api.md
https://github.com/ollama/ollama/blob/main/docs/openai.md
https://github.com/ollama/ollama-python
https://docs.structuredoutputs.xyz/ (constrained decoding concepts)

Change Log

2026-08-26

Initial creation by KB build session.