LLM fundamentals

01The core loop: next-token prediction

An LLM is a function trained to answer one question, extremely well: given this sequence of tokens, what's the probability distribution over the next token? Generation is that function run in a loop:

context → P(next token) → sample one token → append to context → repeat until stop

Everything strange and wonderful about LLMs falls out of this loop:

• Autoregressive. Each token is generated conditioned on everything before it, one at a time. This is why output streams, why latency scales with output length, and why an early wrong token can steer the whole continuation off course.
• No lookup step. No database, no explicit fact store. Knowledge is compressed into the weights during training as statistical structure — and that compression is lossy, which is the root of hallucination (§6).
• "Reasoning" emerges because predicting text written by reasoning humans well requires internal computation that functions like reasoning. Whether to call it that is philosophy; operationally, chain-of-thought works because generating intermediate steps gives the model more serial compute and a scratchpad in its own context (§7).

02Tokens, prefill & decode

Models don't see characters or words — they see tokens: subword chunks from a learned vocabulary (BPE-style, ~50–200k entries). "incident" might be one token; "Sentinel" might be two; rare strings, code, and non-English text fragment into more tokens per character.

• Billing and limits are in tokens. Rough English heuristic: 1 token ≈ ¾ of a word, ~4 characters. JSON and code are token-inefficient — braces, quotes, and whitespace all cost.
• Token-boundary artifacts explain classic failures: strawberry-r-counting and arithmetic weirdness happen partly because the model literally doesn't see letters or digits individually.

Why input tokens are cheap-ish and output tokens are expensive

Latency splits into two phases with completely different shapes:

# All input tokens processed
# in PARALLEL, one pass.
attend(tok₁ … tokₙ)  # at once
fast per token — GPU eats the
whole prompt in one gulp.
Cost ∝ prompt size, but the
wall-clock hit is small.

# Output generated SERIALLY,
# one token per full pass.
while not stop:
    next = sample(P(·|ctx))
    ctx += next   # repeat…
slow — dominates wall-clock.
Each token waits for the last.

03Transformer intuition — attention, no heavy math

The problem: to predict the next token, the model must combine information from anywhere in the context — the function name defined 3,000 tokens ago, the "not" that flips a sentence's meaning. Older architectures (RNNs) passed information through a sequential bottleneck; distant context degraded.

Attention is the answer: for every token position, the model computes how relevant is every other token to me right now, and pulls in a weighted blend of their information. Mechanically:

• Each token emits a query — "what am I looking for?"
• Every token offers a key — "what I'm about" — and a value — "what I carry."
• Relevance = query·key match; the token's updated representation = relevance-weighted sum of values.

It's a soft, learned, content-based lookup over the entire context — done in parallel, not sequentially.

Multi-head, multi-layer: each layer has many attention "heads" learning different relationship types (syntax, coreference — "its" → which noun?, code structure), interleaved with feed-forward layers where much of the stored knowledge seems to live. Stack dozens of layers: early layers resolve local structure, later layers assemble increasingly abstract representations, and the final layer's output is scored against the vocabulary to produce next-token probabilities.

Three consequences worth stating in an interview

1. Attention compares every token with every token → O(n²) in context length. This is why long context is expensive and why context windows were historically small.
2. Parallelism over the sequence is what made training on internet-scale data feasible — the transformer won on trainability, not just capability.
3. During generation, per-token attention results are cached so each new token only computes its own attention — see below.

04Context windows

The context window is the model's entire working memory: system prompt + conversation + retrieved docs + tool definitions + tool results + its own output so far. Nothing outside it exists for the model. There is no persistent memory between calls — "memory" in products is engineering: summarization, retrieval, state you re-inject.

Modern windows are large (128k–1M+), but "fits" ≠ "used well":

• Lost-in-the-middle: retrieval quality across a long context is not uniform — information buried mid-context is recalled worse than content near the start or end. Placement of critical instructions and facts matters.
• Effective vs advertised length: models degrade on complex reasoning over very long contexts well before the hard limit. Long-context benchmarks ("needle in a haystack") measure retrieval, not reasoning-over-everything.
• Cost and latency scale with context — every request re-processes the whole prompt (unless prefix-cached). Stuffing 100 log files into context when 5 matter is paying real money to reduce accuracy — the practical argument for retrieval and context curation even when everything technically fits.
• Agent-specific pressure: agent loops accumulate context — every tool result appends. Long-running agents need context management: truncating stale tool output, summarizing history, or resetting with distilled state.

05Sampling: temperature, top-p, and friends

The model outputs a probability distribution; sampling strategy decides how you pick from it.

• Temperature rescales the distribution before sampling. T→0: always take the argmax (greedy) — maximally deterministic-ish, best for extraction/classification/structured output. T≈0.7–1: sample proportionally — diversity for writing and brainstorming. T>1: flatten toward uniform — incoherence risk. Intuition: temperature is how much probability mass you're willing to give to non-top choices.
• Top-p (nucleus): sample only from the smallest set of tokens whose cumulative probability ≥ p (e.g., 0.9). Adaptive cutoff: when the model is confident the nucleus is tiny; when uncertain, it's broad. Generally set temperature or top-p, not both aggressively.
• Top-k: fixed cutoff — only the k most likely tokens. Cruder than top-p.

06Why hallucination happens — mechanism, not apology

Four causes, each with the engineering response an interviewer actually wants to hear:

07Prompt engineering that actually matters

Skip the folk magic — these are the techniques with reliable effect, and why they work in terms of §1:

System prompts

A privileged instruction channel, weighted by training to dominate user turns. Use for: role/scope, hard constraints, output contract, tool-use policy. Treat it as versioned code — it is your app's behavior spec: review changes, eval before shipping. The instruction hierarchy (system > developer > user) is also the first line of prompt-injection defense — while noting honestly that it's probabilistic, not a security boundary.

Few-shot examples

2–5 input→output demonstrations. Works via in-context learning: you're conditioning the distribution on a pattern to continue. The high-leverage detail: examples beat descriptions for format and edge-case behavior — show the tricky case (an ambiguous log line and its correct classification), because the model imitates examples more faithfully than it follows prose. Failure mode: examples that are all one class bias outputs toward that class.

Chain-of-thought

"Think step by step" / structured reasoning before the answer. Why it works: each generated token is extra serial computation, and intermediate results written into context become available to condition on — a scratchpad. Matters for math, multi-hop logic, tricky classification. Modern wrinkle: reasoning models (o-series, extended thinking) internalize this — you buy reasoning with a token budget instead of prompting for it, and pay in latency/cost, so route only hard tasks to them.

Structured output

For machine-consumed responses (everything in a pipeline like Sentinel): define the schema explicitly; prefer native structured-output / JSON modes or tool-calling with a schema over "please return JSON" — constrained decoding masks invalid tokens, guaranteeing syntax though not semantic correctness. Still validate with Pydantic and design a repair/retry path. And order the schema deliberately: ask for reasoning fields before conclusion fields — generation is left-to-right, so a "verdict" field generated first is decided before the "evidence" field is written.

Also load-bearing

• Put long documents before instructions — recency helps instruction-following.
• Use delimiters / XML tags to separate untrusted data from instructions.
• Tell the model what to do, not only what not to do.
• Give it an explicit out ("if the logs don't contain enough evidence, say so") — measurably reduces forced confabulation.

08Model landscape awareness & routing

What's expected: not leaderboard trivia, but a decision framework plus a rough map.

Frontier APIs vs open-weights

The tiering pattern — the actually-useful knowledge

The engineering move is routing: match each task to the cheapest tier that passes your evals — with evals as the referee and fallback across tiers/providers for resilience. Second-order factors that decide real selections: tool-calling reliability, structured-output support, context length, rate limits, latency SLOs, data-privacy terms — often more decisive than benchmark deltas.

09Rapid-fire interview Q&A

10Self-test