LLM Introduction - What Large Language Models Actually Are
Status: Active
Last Updated: 2026-08-14
Category: AImL - Phase 1: LLM Fundamentals
Prerequisites: None
Time: 2 hours
Tags: llm, transformers, tokens, parameters, cloud-vs-local, privacy, fundamentals
Summary
A plain-language introduction to how large language models work: what a token is, what "parameters" mean, why transformers made this all possible, and the real trade-offs between paying for cloud AI APIs versus running models on your own hardware. This is the conceptual foundation for every other lesson in this course.
🎯 What You'll Learn
- ✅ Explain, at a working level, what an LLM does when it answers a prompt
- ✅ Describe the transformer architecture without needing linear algebra
- ✅ Understand what model size (7B, 13B, 70B) actually measures
- ✅ Compare cloud APIs vs local inference across cost, privacy, latency, and control
- ✅ Make an informed decision about where your prompts and data should go
1. What Is a Language Model, Really?
Strip away the marketing and an LLM does exactly one thing:
Given a sequence of tokens → predict the next token → repeat
That's it. Everything that looks like reasoning, conversation, or coding skill is an emergent property of doing next-token prediction extremely well over enormous amounts of text.
The Autoregressive Loop
When you send the prompt "The capital of France is" to a model, here's the actual loop:
Input: "The capital of France is"
Step 1: predict next token → " Paris"
Input: "The capital of France is Paris"
Step 2: predict next token → "." (or continue...)
...repeat until the model emits a stop token or hits a limit
What Happens: Each generated token gets appended to the input, and the whole thing (prompt + everything generated so far) is fed back in for the next prediction. This is why long generations get slower — there is more context to process each round — and why models sometimes "lose the thread" late in very long outputs.
Tokens, Not Words
Models don't see words; they see tokens — chunks of text from a fixed vocabulary (typically 30k–250k entries). A rough rule of thumb for English:
| Text | Approximate tokens |
|---|---|
hello |
1 |
unbelievable |
3 (un, believ, able) |
| This sentence. | ~5 |
| One page of text | ~500 |
| This lesson | ~6,000 |
Why this matters practically:
- Cost/limits are measured in tokens, not words or characters
- Context windows are token budgets shared between your prompt and the model's answer
- Unusual languages, code, or heavy punctuation consume more tokens per idea
You can see tokenization yourself at any tokenizer playground (e.g., OpenAI's or Hugging Face's online tokenizers).
2. How LLMs Work (Simplified)
You do not need the math to operate AI infrastructure, but you need the shape of it.
2.1 Training vs Inference
Two completely different phases get conflated constantly:
| Phase | What happens | Cost | Frequency |
|---|---|---|---|
| Training | Read trillions of tokens, adjust billions of weights via gradient descent | Months of GPU-cluster time | Once per model release |
| Inference | Run the frozen weights on new input to predict tokens | Milliseconds-to-seconds per response | Every time anyone uses the model |
Key insight for self-hosters: you only ever run inference locally. Training (or fine-tuning, covered later in fine-tuning-basics) needs far more hardware than almost any homelab has.
2.2 The Transformer Architecture
Introduced in the 2017 paper "Attention Is All You Need". The pieces that matter to you as an operator:
Embeddings Every input token is converted into a vector — a list of numbers (hundreds to thousands of dimensions) that encodes meaning. Similar meanings land near each other in this space. This exact mechanism powers vector databases later in this course (embeddings-explained).
Attention
For each token being predicted, attention layers let the model look back at all previous tokens and decide which ones matter right now. In "The cat sat on the mat because it was tired," attention connects it → cat.
This is also the source of the practical constraint you'll fight constantly:
- Attention cost grows with sequence length → long contexts are expensive
- The context window is a hard architectural limit per model
Layers Transformers stack these attention + processing blocks dozens of times (32 layers in an 8B-class model, ~80+ in 70B-class). Early layers capture syntax; deeper layers capture abstract semantics.
Feed-forward networks + residual connections Per-token computation and the wiring that lets deep networks train stably. You don't tune any of this; just know that parameter counts mostly live here.
2.3 From Raw Prediction to Assistant
A raw next-token predictor is not a helpful assistant. Modern models get shaped through:
- Pre-training — predict the next token over internet-scale text
- Instruction tuning — fine-tune on examples of instructions and good responses
- Preference alignment (RLHF/DPO) — humans rank outputs; the model is nudged toward preferred behavior
Practical consequence: two models with identical sizes can behave very differently depending on stages 2–3. Model choice is about more than parameter count (model-selection-guide).
2.4 Sampling: Why Answers Vary
At each step the model produces a probability distribution over the vocabulary, and something has to pick the next token:
| Setting | Meaning | Typical use |
|---|---|---|
temperature |
Scales randomness. 0 ≈ deterministic argmax, higher = more adventurous | 0–0.3 factual/extraction, 0.7–1.0 creative |
top_p |
Only sample from tokens covering top-p cumulative probability | Usually leave at default |
top_k |
Only sample from k most likely tokens | Alternative to top_p |
# Same prompt, temperature 0: nearly identical output every run
ollama run llama3.1:8b --temp 0 "Name the capital of France."
# Temperature 1: varied phrasing run to run
ollama run llama3.1:8b --temp 1 "Write one surprising sentence about Paris."
What Happens: Low temperature collapses the distribution toward the single most likely token — good for facts, bad for creativity. High temperature spreads probability mass — good for brainstorming, dangerous for math.
3. Parameters and Model Sizes
"Parameters" = the learned weights inside the network. More parameters generally means more capacity to store patterns and knowledge — at proportional cost in memory and speed.
3.1 Reading Model Names
llama3.1:8b decodes as family Llama, version 3.1, roughly 8 billion parameters.
| Size class | Examples | RAM/VRAM needed (Q4 quantized) | Character |
|---|---|---|---|
| 1–4B | Phi-3-mini, Gemma 2B, Qwen2.5 3B | 2–6 GB | Fast, surprisingly capable at narrow tasks, weak at complex reasoning |
| 7–9B | Llama 3.1 8B, Mistral 7B | 6–10 GB | The homelab sweet spot — good quality, interactive speed |
| 12–15B | Qwen 14B, Phi-4 14B | 10–16 GB | Noticeably better reasoning, still consumer-feasible |
| 27–35B | Gemma 27B, Qwen 32B | 18–24 GB | High quality; wants a 24GB GPU or fast CPU+RAM |
| 70B+ | Llama 3.3 70B | 40–48 GB | Near-frontier quality; multi-GPU or heavy quantization territory |
3.2 The Scaling Intuition
What tends to improve with size:
- Factual recall breadth
- Multi-step reasoning
- Instruction following nuance
- Resistance to obvious traps
What often does NOT improve:
- Speed (gets worse)
- Style — a small model well-tuned for chat can feel better than a bigger generic one
- Domain narrow tasks — a 3B code model can beat a general 70B at simple completions
3.3 Quantization Sneak Peek
Raw weights are stored as 16-bit floats. Quantization compresses them to 4-bit integers (Q4), cutting memory ~4× for modest quality loss. That's why an 8B model fits in ~5GB instead of ~16GB. Full treatment in model-quantization; for now, know that most "b" sizes you download via Ollama are already Q4 by default.
4. Cloud vs Local LLMs
The central infrastructure decision of this course.
4.1 Cloud APIs
OpenAI, Anthropic, Google, Mistral, etc. You POST a prompt over HTTPS; they run the giant model; you pay per million tokens.
Strengths:
- Frontier quality — nothing you can self-host matches the largest closed models
- Zero hardware — works from a laptop
- Elastic scale — no capacity planning for spikes
Weaknesses:
- Per-token cost compounds brutally at volume
- Your prompts (and attached data) transit and reside on someone else's systems
- Rate limits, outages, deprecations, price changes — all outside your control
- Network latency on every call
4.2 Local Models (Ollama et al.)
You download open-weight models (Llama, Mistral, Qwen, Gemma...) and run inference on your own metal with tools like Ollama.
Strengths:
- Privacy: prompts never leave your machine — decisive for personal notes, client data, medical/legal text
- Zero marginal cost per token after hardware
- No rate limits, no deprecations, works offline
- Deterministic environment — the model you tested is the model in production
Weaknesses:
- Quality ceiling below frontier cloud models (though closing yearly)
- Hardware-bound: RAM/VRAM limits which models you can even load
- You own uptime, updates, and monitoring
- Slower on CPU-only boxes
4.3 Decision Table
| Criterion | Cloud API | Local (Ollama) |
|---|---|---|
| Data sensitivity (personal/client/regulated) | ⚠️ Review contracts & regions | ✅ Never leaves the box |
| Upfront cost | None | $0 (existing PC) → $2k+ (GPU box) |
| Ongoing cost | Per token, forever | Electricity only |
| Peak quality | ✅ Best available | Good (7B–70B open models) |
| Latency (first token) | 200ms–1s + network | Local GPU: <150ms; CPU: seconds |
| Works offline / air-gapped | ❌ | ✅ |
| Rate limits / vendor lock-in | Yes | No |
| Ops burden | Minimal | Yours (updates, storage, monitoring) |
4.4 Hybrid Is Normal
Real deployments mix both:
- Local model for high-volume, sensitive, or routine work (classification, summarization, drafting)
- Cloud API for occasional frontier-reasoning tasks
- A routing layer sends each request to the right tier (see agent-frameworks and rag-production-patterns)
5. Privacy Considerations
Even if you choose cloud APIs somewhere in your stack, do it deliberately:
Questions to ask before sending data anywhere external:
- Is my prompt retained? For how long? Used for training?
- Where geographically is it processed/stored?
- Does the provider offer a zero-retention or business tier?
- Would I be comfortable if this exact text appeared in a future training corpus?
Rules of thumb for self-hosted setups:
- Treat anything typed into a third-party web chat UI as public
- Secrets, API keys, customer PII, unreleased code: local models only, or redact first
- Even locally, remember RAG indexes concentrate your sensitive documents in one searchable place — protect the vector DB like any database (see qdrant-setup)
The self-hosting thesis of this course: for the vast majority of everyday assistant tasks — summarizing, drafting, classifying, Q&A over your own documents — a local 7–14B model is genuinely good enough, and the privacy win is total.
6. Hands-On: Feel It For Yourself
No setup required yet (that's ollama-setup), but internalize these facts:
Fact 1 — Generation is sequential. Watch any streaming LLM output: words appear one chunk at a time because each token literally depends on all previous ones. There is no way to "parallelize" a single generation. (Batching multiple independent requests is different — gpu-optimization.)
Fact 2 — Context is finite and shared. Whatever the model "remembers" within one conversation must fit in its context window along with your question. Long chats silently push early content out (truncation strategies in context-and-tokens).
Fact 3 — The model has no memory of you between requests. Every API call is independent. "Remembering" anything requires you to resend relevant history or store/retrieve it externally — which is precisely the problem RAG solves in Phase 4.
🛠️ Troubleshooting & Common Issues (Conceptual Level)
| Symptom | Root cause | Fix direction |
|---|---|---|
| Model states false facts confidently | Hallucination — it's a predictor, not a database | Ground it in retrieved documents (RAG), lower temperature, ask for citations |
| Answer ignores instructions buried mid-prompt | Attention dilution in very long prompts | Put critical instructions at start AND end; shorten context |
| Same question gives different answers | Sampling temperature > 0 | Set temperature 0 for reproducibility |
| Output cut off mid-sentence | Hit max_tokens or context limit |
Raise limits; check context-and-tokens |
| Model claims a cutoff date for recent events | Training data horizon | Provide current facts in the prompt (again: RAG) |
| Slow responses | Model too big for hardware, or CPU-only inference | Smaller/quantized model; GPU; see gpu-optimization |
🔗 Related
- Next: ollama-setup — install Ollama and run your first model
- Also next: model-selection-guide, ollama-cli-basics
- Deep dives referenced above: embeddings-explained, context-and-tokens, model-quantization
- Course prerequisites: kb/basics/, containers/docker-basics
- Philosophy: TELOS — self-hosted, private-by-default infrastructure
📝 Change Log
- 2026-08-14 — Initial publication. Covers README outline: how LLMs work (simplified), transformer architecture, parameters/model sizes, cloud vs local, privacy considerations.