fogserv.cloud fogserv.cloud

RAG Pipeline - Retrieval Augmented Generation with Ollama and Qdrant

Status: Active
Last Updated: 2026-08-26
Category: AI/ML - Phase 4: RAG Systems
Prerequisites: embeddings-vector-db, ollama-api
Time: 2-3 hours
Tags: rag, retrieval-augmented-generation, qdrant, ollama, chunking, embeddings, llm, python

Summary

RAG (Retrieval Augmented Generation) fixes the two biggest LLM weaknesses — stale knowledge and confident hallucination — by retrieving relevant chunks of your own documents at query time and handing them to the model as context. This guide builds a complete minimal RAG pipeline in Python: load a document, chunk it with sensible size/overlap, embed chunks into Qdrant, retrieve top matches for a question, assemble a grounded prompt, and generate an answer with a local Ollama chat model — plus evaluation tips and the failure modes that quietly wreck answer quality.

🎯 What You'll Learn

By the end of this article, you'll be able to:

Table of Contents

  1. What RAG Is and Why
  2. Document Loading
  3. Chunking Strategies: Size and Overlap
  4. Embedding and Storing Chunks in Qdrant
  5. Retrieval and Prompt Assembly
  6. End-to-End Minimal RAG Script
  7. Evaluation Tips
  8. Failure Modes
  9. Troubleshooting & Common Pitfalls
  10. Sources & Related
  11. Change Log

1. What RAG Is and Why

An LLM only knows what was in its training data plus what you put in its prompt. RAG closes that gap:

Question → embed → search Qdrant → top-k chunks
                                        ↓
        LLM ← "Answer ONLY from this context: <chunks> Question"
                                        ↓
                                  Grounded answer (+ citations)

Why RAG instead of fine-tuning?

Fine-tuning RAG
Adds new facts/knowledge Poorly (teaches style, not facts) ✅ Exactly what it's for
Update frequency Retrain per update Re-index a doc in seconds
Citations / provenance None ✅ Chunk-level sources
Cost & hardware GPU training runs CPU-only local stack
Hallucination control Weak ✅ "answer from context" constraint

You already have all the primitives from Phase 3 (embeddings-vector-db). RAG just wires them into a loop around a chat model.

2. Document Loading

Start simple — plain text and Markdown cover most home-lab use:

from pathlib import Path

def load_text(path: str) -> str:
    return Path(path).read_text(encoding="utf-8")

For real corpora, add loaders incrementally: PDF (pypdf), HTML (beautifulsoup4 + html2text), or a directory walk over *.md storing {text, source} records. Keep source paths in payloads from day one — answers without provenance are hard to trust.

3. Chunking Strategies: Size and Overlap

Embedding models cap input length (~2K tokens for nomic-embed-text), so long documents must be split. Chunking is the single highest-leverage knob in a RAG pipeline: too big dilutes the semantic signal (one vector can't represent 10 unrelated topics); too small loses surrounding context.

Rules of thumb:

A simple word-overlap chunker (good enough to start; swap in a tokenizer-based one later):

def chunk_text(text: str, size_words=350, overlap_words=60):
    """Split text into overlapping word-window chunks."""
    words = text.split()
    step = size_words - overlap_words
    return [
        " ".join(words[i:i + size_words])
        for i in range(0, len(words), step)
        if len(words[i:i + size_words]) > 20   # drop tiny tail fragments
    ]

Alternatives worth knowing: recursive character splitting (LangChain's RecursiveCharacterTextSplitter), sentence-aware splitting, and semantic chunking (split where embedding similarity between consecutive sentences drops).

4. Embedding and Storing Chunks in Qdrant

import ollama
from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams

MODEL = "nomic-embed-text"      # 768 dims, 2K context
client = QdrantClient(url="http://localhost:6333")

client.recreate_collection(
    collection_name="docs",
    vectors_config=VectorParams(size=768, distance=Distance.COSINE),
)

def index_document(path: str):
    text = load_text(path)
    chunks = chunk_text(text)
    vectors = ollama.embed(model=MODEL, input=chunks)["embeddings"]
    client.upsert(
        collection_name="docs",
        points=[
            {
                "id": f"{path}-{i}",
                "vector": vec,
                "payload": {"text": chunk, "source": path, "chunk": i},
            }
            for i, (chunk, vec) in enumerate(zip(chunks, vectors))
        ],
    )
    print(f"indexed {len(chunks)} chunks from {path}")

Batching all chunk embeddings into one ollama.embed(input=[...]) call is dramatically faster than one request per chunk.

5. Retrieval and Prompt Assembly

Retrieval mirrors Phase 3: embed the question, query_points, take top-k (start with k=3–5).

Prompt assembly is where hallucination is won or lost. Three rules:

  1. Constrain: instruct the model to answer only from the provided context.
  2. Admit ignorance: explicitly allow "I don't know" when the context doesn't contain the answer.
  3. Attribute: ask the model to cite which sources it used.
PROMPT_TEMPLATE = """You are a helpful assistant. Answer the user's question
using ONLY the context below. If the context does not contain enough
information, say "I don't know" - do not invent facts.

Context:
{context}

Sources: {sources}

Question: {question}

Answer:"""

6. End-to-End Minimal RAG Script

Complete working script — index a Markdown file, then answer questions from it. Requires: Ollama running with ollama pull nomic-embed-text and any chat model (e.g. llama3.1 or qwen2.5), plus Qdrant from Phase 3.

pip install ollama qdrant-client

"""rag.py — minimal end-to-end RAG: Ollama embeddings + Qdrant + Ollama chat."""
import sys
import ollama
from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams
from pathlib import Path

EMBED_MODEL = "nomic-embed-text"     # 768 dims
CHAT_MODEL  = "llama3.1"             # any Ollama chat model works
COLLECTION  = "docs"

client = QdrantClient(url="http://localhost:6333")

def ensure_collection():
    if not client.collection_exists(COLLECTION):
        client.create_collection(
            COLLECTION,
            vectors_config=VectorParams(size=768, distance=Distance.COSINE),
        )

def chunk_text(text, size_words=350, overlap_words=60):
    words, step = text.split(), size_words - overlap_words
    return [" ".join(words[i:i+size_words]) for i in range(0, len(words), step)
            if len(words[i:i+size_words]) > 20]

def index(paths):
    points, offset = [], 0
    for p in paths:
        for i, chunk in enumerate(chunk_text(Path(p).read_text())):
            vec = ollama.embed(model=EMBED_MODEL, input=chunk)["embeddings"][0]
            points.append({"id": offset, "vector": vec,
                           "payload": {"text": chunk, "source": p}})
            offset += 1
    client.upsert(COLLECTION, points)
    print(f"Indexed {offset} chunks.")

PROMPT = """Answer ONLY from the context below. If it's not there, say "I don't know".

Context:
{context}

Question: {question}
Answer:"""

def ask(question, k=4):
    qv = ollama.embed(model=EMBED_MODEL, input=question)["embeddings"][0]
    hits = client.query_points(COLLECTION, query=qv, limit=k).points
    context = "\n\n---\n\n".join(h.payload["text"] for h in hits)
    print("Sources:", ", ".join({h.payload["source"] for h in hits}))
    reply = ollama.chat(model=CHAT_MODEL, messages=[
        {"role": "user", "content": PROMPT.format(context=context, question=question)}
    ])
    print(reply["message"]["content"])

if __name__ == "__main__":
    ensure_collection()
    if sys.argv[1] == "index":
        index(sys.argv[2:])
    else:
        ask(" ".join(sys.argv[1:]))

Usage:

python rag.py index ./kb/aiml/*.md
python rag.py how do I choose a chunk size?

7. Evaluation Tips

Before adding features, measure. Cheap, effective checks:

Upgrade path once basics work: hybrid search (dense + keyword/BM25 fusion), reranking the top-k with a cross-encoder, and metadata filtering by source.

8. Failure Modes

Failure Symptom Fix
Bad chunking Right topic, wrong snippet; answers cut mid-thought Adjust size/overlap; split on headings first
Retrieval miss Model says "I don't know" despite the answer being in docs Increase k; check chunking; try a better embedding model
Context stuffing Answer degrades with many chunks; slow Reduce k; filter low-score hits above a threshold
Hallucination despite context Confident answer contradicting sources Strengthen prompt constraints; switch to a stronger chat model
Model/context mismatch Vector errors or nonsense results after changing models Re-index the whole collection whenever the embedding model changes
Stale index Answers reflect deleted/outdated docs Delete old points on re-index; version collections

Troubleshooting & Common Pitfalls

Click to expand
  • Everything indexed but every answer is wrong → test retrieval alone first (print top-k chunks before invoking the LLM). In ~80% of broken pipelines, retrieval is the culprit, not generation.
  • size mismatch on upsert/search → collection created with a different dimension than current model. Drop and recreate the collection.
  • Slow indexing → embedding chunks one-by-one via REST. Batch with ollama.embed(input=[list]).
  • Answers ignore the "only from context" rule → small local chat models comply loosely; put constraints at the start of the prompt, keep them short and imperative.
  • Chunks exceed the 2K-token embed window → nomic-embed-text silently truncates, producing weak vectors. Count tokens, not characters.
  • Duplicate content in answers → re-ran indexing without clearing; upsert with stable deterministic IDs (e.g., path-chunkN) makes re-indexes idempotent.

Sources & Related

Web sources consulted during research:

Related knowledge-base articles:

Change Log

Choose Theme

Your selection is saved locally.

Neural Cacophony
Aperture v2
Flux v1
Mosaic Chaos
Nexus v1
Nexus Zest
Prism v2
Synapse