Module 2 — How LLMs Work

In this module

• Understand what a token actually is and why it drives every limit and cost you'll hit.
• Know the context window as the model's short-term memory — and how to work within it.
• Use temperature deliberately: when to set it low, when to set it high.
• Understand why LLMs hallucinate and what that means for how you design around it.
• Get a working intuition for embeddings before we use them for RAG.
• Understand statelessness: why the model forgets between calls and how to handle it.

Time: ~30 min · Lab: Lab 2 — First API Call

Why this matters

You can use an LLM without knowing anything in this module — until something goes wrong. The model confidently invents a fact. The chatbot forgets a detail from three turns ago. Your careful prompt suddenly produces wildly inconsistent outputs. Every one of these surprises has a direct, mechanical explanation rooted in how LLMs work at runtime. This module gives you those explanations — not as theory, but as actionable mental models you can apply every time you open an API playground or debug a production prompt.

Tokens — the unit the model thinks in

Tokens are not words. They are the fragments of text your prompt and the model's response are chopped into before any processing happens. A token is roughly 3–4 characters of English: the word unbelievable becomes three tokens (un, believ, able). A short word like I is a single token. A 1,000-word document is about 1,300 tokens.

Why does this matter practically?

• Billing is per token, not per word. A 10-page PDF is probably 5,000–6,000 tokens.
• Context limits are in tokens. "128K context window" means 128,000 tokens total — prompt, history, and response combined.
• Speed is measured in tokens per second. A fast model doing 100 tokens/s generates roughly 75 words per second.
• Non-English languages tokenize less efficiently with most current vocabularies — a Chinese or Arabic sentence might use twice as many tokens as the equivalent English.

The developer instinct to develop: whenever you see a limit or a cost you don't understand, ask "how many tokens is that?" Paste your text into Tiktokenizer and count.

How an LLM generates text — the next-token autocomplete loop:

flowchart LR
  Prompt["Prompt Tokens"] --> Model["LLM"] --> Next["Predict Next Token"] --> Append["Append to Output"] --> Model

Context window — the model's short-term memory

The context window is the total number of tokens the model can consider in a single call: your system prompt, the conversation history, any documents you pasted in, and the model's response all share this fixed budget. When you exceed the limit, the API either refuses the call or silently drops the oldest content.

Think of the context window as the call-stack size of the conversation. Just as a call stack keeps only the frames currently in scope — and pops old frames when they run out — the context window keeps only what you sent in this request. Nothing persists automatically between calls. Give the model 128K tokens of budget and you can include an entire codebase in one shot. Give it 8K and you need to be surgical about what you include.

Windows have grown fast — from a few thousand tokens to, at the frontier, around a million (the exact numbers move every few months, so treat any figure as a snapshot). Bigger windows are not always better — cost scales with tokens processed, models can lose focus when given too much irrelevant context, and usable quality often fades well before the stated limit. Include what's relevant; leave out what's not.

Mental model

Imagine you're on a phone call and your notes only hold one page. You can write down as much context as fits on that page before the call starts, but once the page is full, writing more means erasing something old. The context window is that page. You — or your application code — are responsible for deciding what stays on it.

Temperature — the randomness dial

Temperature is a parameter — typically on a 0 to 2 scale, depending on the provider — that controls how randomly the model picks its next token.

At temperature 0, the model always picks the single most probable next token. The output is deterministic: run the same prompt twice and get identical results. Use this for structured data extraction, JSON output, code generation where correctness matters, and anything you want to unit-test.

At temperature 1 (or above), the model samples from a probability distribution over many candidate tokens. The output is varied: run the same prompt twice and get meaningfully different results. Use this for brainstorming, creative writing, generating diverse test data, and any task where variety is the goal.

Most real prompts live in the range 0.2–0.7. A customer-support bot should be closer to 0; a creative-writing assistant closer to 0.8. A code generator should default to 0 and expose a knob only if the user explicitly wants varied alternatives.

The mistake to avoid: leaving temperature at its default (often 0.7 or 1.0) for tasks that require consistent, parseable output. Nothing breaks a downstream JSON parser quite like a model that sometimes adds an extra sentence before the opening brace.

The same prompt at two temperatures — the model's choice of next word:

Why LLMs hallucinate

Hallucination — a model stating false things confidently — is not a bug waiting to be fixed. It is a direct consequence of the architecture.

LLMs are next-token predictors. At every step, they produce the token that statistically fits best given the tokens so far. They do not look up facts in a database; they do not verify claims against the real world; they do not have a "true/false" circuit. They generate text that looks like what an authoritative answer would look like — because that is what appeared adjacent to similar questions in training data.

This means a model will:

• Invent plausible-sounding citations — correct author name style, plausible journal name, wrong paper.
• Hallucinate API method names — the name it generates looks like the library's naming convention; the method doesn't exist.
• Confabulate recent events — events after the model's training cutoff simply don't exist in its weights; it will still produce something fluent.

The engineering lesson: treat every factual claim from an LLM as a hypothesis to verify, not a fact to cite. Design your system accordingly — use RAG to ground the model in documents you control, use structured output to constrain it to shapes you can validate, and use evals to catch regressions in factual accuracy.

Worked example — one word's journey from text to vectors:

(numbers illustrative)

The tokenizer splits unbelievable into three sub-word tokens — un, believ, able. Each token maps to an integer ID (its index in the model's vocabulary), and each ID is looked up in an embedding table to retrieve its embedding vector — a list of numbers that encodes that token's meaning. This is what actually flows into the model; the raw text you typed is transformed into these vectors before any prediction happens.

Embeddings — a brief introduction

An embedding is a list of numbers that encodes the meaning of a piece of text. The key property: texts with similar meanings end up close together in vector space. "The server is down" and "the service is unavailable" map to nearby points; "My cat knocks glasses off tables" maps to a very different region.

Any unstructured data — text, an image, audio — turns into a list of numbers:

Those numbers are coordinates on a map of meaning:

In plain words

Imagine a library where books on similar topics are shelved next to each other — cookbooks in one aisle, car manuals in another. An embedding is the model's way of giving every word or sentence a shelf location, so "close on the shelf" means "close in meaning." That's why "the server is down" lands right next to "the service is unavailable," even though they share no words.

You don't need to understand the math here. The practical consequence: embeddings let you search by meaning rather than by exact keyword match. Later in this course we'll use them as the foundation for RAG — fetching the most semantically relevant chunks from your documents and injecting them into the context window before the model answers.

For now, file this as: an embedding is the model's internal way of representing meaning as a position in space, and that position is searchable.

Statelessness — every call is fresh

Stateless means the LLM API remembers nothing between calls. Each request is processed in complete isolation. The model has no memory of previous conversations, no persistent session, no ability to "remember what you told it last Tuesday."

Think of an LLM API as a pure function:

response = llm(system_prompt + message_history + new_message)

The function takes everything relevant as input and returns output. If you want the model to remember the first three turns of a conversation, you must include those three turns in the message_history you pass on turn four. The application layer — your code — owns state management.

This is where the system prompt comes in. It is a special, persistent instruction block sent at the top of every call, before the user's message. It's the closest thing to configuration the API exposes: "You are a helpful customer-support agent for Acme Corp. Reply in British English. Never discuss competitor pricing." Think of it as the constructor arguments for the model's behaviour in your application — they're passed on every call, and they frame every response that follows.

The statelessness property has two important engineering implications:

1. Cost grows with history. To maintain conversation context over 10 turns, you resend all 10 turns every time. A 30-minute chat can accumulate thousands of tokens of history. Plan for truncation or summarisation.
2. Determinism is achievable. Because the model has no hidden session state, two identical requests (same model, same temperature=0, same messages) produce identical responses. This makes LLM behaviour more testable than it might appear.

KV caching — why generation stays fast

Look at the next-token autocomplete loop above: the model generates one token at a time, then feeds it back as input for the next step. Without any optimisation, each step would mean re-processing the entire conversation history from scratch — getting slower with every token added.

The KV cache is how the model avoids that. Internally, each transformer layer computes keys and values for every token it processes. Once computed, these are stored in a cache so the next generation step can reuse them instead of recomputing. Only the brand-new token needs fresh computation each step. That is why streaming output arrives at a steady pace rather than slowing down as the response grows — and why longer contexts (bigger caches) consume more GPU memory.

With vs without a KV cache, when generating the next token:

In plain words

It's like writing a long letter. Without a KV cache, you'd re-read the entire letter from the top before adding each new sentence — slower and slower as it grows. With the cache, you just glance at your running notes and write the next sentence. That's why streaming output keeps a steady pace instead of crawling near the end — and why a longer conversation eats more memory (the notes get bigger).

Beyond text — multimodal models

Most models you'll use today are multimodal: they accept more than text — images, audio, sometimes video — and some generate images too. Under the hood it's the same move as embeddings: a picture or an audio clip is turned into vectors the model processes right alongside text. Practically, that means you can hand a model a screenshot, a chart, a photo of a receipt, or a UI mockup and ask questions about it — no separate OCR pipeline required. When you evaluate a model, "multimodal" is now a capability to check for, not an exotic extra.

Reasoning models — paying the model to think first

Everything above describes a model that answers in one fast pass: tokens in, tokens straight out. A newer class — reasoning models (often called "thinking" models) — adds a deliberate step before the visible answer. The model first generates a private chain of reasoning — trying approaches, checking itself, backtracking — and only then writes its reply. Examples: OpenAI's o-series and DeepSeek R1.

The idea has a name: test-time compute — spend more computation at the moment you ask, not only during training. Letting a model think longer on a hard problem can beat simply reaching for a bigger model.

What this means for you as a developer:

• They cost more and respond slower. The thinking is generated as reasoning tokens you pay for, even when you never see them — a reasoning model can burn several times the tokens of a standard call.
• You prompt them differently. "Think step by step" is largely redundant — the model already reasons internally. Give it the goal and the constraints; don't micromanage the steps.
• Use them selectively. For multi-step maths, logic, planning, hard debugging, and tricky analysis, a reasoning model is often worth the cost. For classification, extraction, formatting, and simple Q&A, a fast standard model is cheaper and just as good.

In plain words

A standard model answers instantly, off the top of its head. A reasoning model says "give me a minute," works it out on scratch paper, then tells you the answer — slower and pricier, but far more reliable on genuinely hard questions. The routing rule mirrors model size: default to a fast model; escalate to a reasoning model only when the task truly needs multi-step thinking.

Hands-on

Open Lab 2 — First API Call. You'll make your first raw API call, observe the token counts returned in the response, and experiment with temperature to see how output variance changes.

Go deeper

• Tiktokenizer — Interactive Tokenizer Playground — Paste any text and see it broken into colour-coded tokens in real time; a two-minute session here builds more token intuition than an hour of reading.

• Guide to LLM Hallucinations — Lakera — A practical, developer-oriented breakdown of what hallucination is, why it happens, and concrete mitigation strategies (RAG, grounding, evals); the best single page on the topic.

• Are LLMs Stateless? Architecture, Implications and Solutions — Atlan — Clear explanation of why LLMs are stateless, what that means for application design, and how to handle conversation memory — written for engineers, not researchers.

• Context Window Management for LLM Apps — Redis — A practical developer guide covering truncation, summarisation, and retrieval strategies for keeping context windows under control in production applications.

Jargon recap

token · context window · temperature · hallucination · embedding · KV cache · stateless · system prompt · multimodal model · reasoning model · test-time compute · reasoning tokens

Check yourself

Try to answer before expanding — active recall is what makes it stick.

Why are tokens (not words) the unit that drives cost and limits?

Text is chopped into tokens — roughly 3–4 characters each — before any processing, so unbelievable is three tokens. Billing, context windows, and speed are all measured in tokens, and non-English text often uses far more tokens for the same meaning.

Why does temperature 0 matter, and when would you raise it?

At temperature 0 the model always picks the most probable next token, so output is deterministic and repeatable — use it for extraction, JSON, code, and anything you want to unit-test. Raise it (0.7–1.0) for brainstorming, creative writing, or varied output where variety is the goal.

Why do LLMs hallucinate, and what does that imply for your design?

They are next-token predictors — they generate text that looks like an authoritative answer, with no fact-checking circuit. So treat every factual claim as a hypothesis to verify, and design around it with RAG (grounding), structured output, and evals.

What does it mean that LLMs are stateless, and who manages conversation memory?

Each API call is processed in isolation — the model remembers nothing between calls; it is effectively a pure function of everything you pass in. Your application code owns state: you must resend the relevant history every turn (which is why cost grows with conversation length).

Key takeaways

• Tokens are the model's atomic unit — billing, context limits, and speed are all measured in tokens; 1,000 English words ≈ 1,300 tokens.
• The context window is the model's entire short-term memory for one call; your application code is responsible for deciding what goes in it.
• Temperature 0 = deterministic and repeatable; higher = varied and creative — match the dial to the task, not to the default.
• Hallucination is structural: the model predicts plausible tokens, not verified facts; always treat factual claims as hypotheses.
• LLMs are pure functions — they are completely stateless; the system prompt is your per-call configuration, and conversation history is your responsibility to pass in.

---

Next: Prompt Engineering →