LLM Ops & Evals

01The production frame

An LLM in production is a dependency with unusual properties: non-deterministic, slow (seconds), expensive (per token), rate-limited, and silently version-shifting under you. LLM-ops is ordinary production discipline — retries, caching, observability, testing — re-derived for a component with those properties.

02Serving: streaming and token budgeting

TTFT vs decode — the two halves of latency

Total latency ≈ prefill (prompt processing) + decode (output generation, serial per token). A 500-token answer at ~50 tok/s is ~10s — unacceptable as a blank spinner, fine as a stream. Streaming doesn't reduce total latency; it collapses perceived latency to TTFT.

# perceived responsiveness
dominated by:
  prefill (prompt processing)
  + queueing at the provider

improve with:
  prefix caching (skip re-prefill)
  smaller context
  faster model tier
  less queue contention

# sustained generation speed
serial: one token at a time
500 tok @ ~50 tok/s ≈ 10s

improve with:
  shorter outputs (max_tokens,
    concise-output instructions)
  faster model tier
  streaming (hides it, doesn't
    shrink it)

Engineering realities worth naming:

• Transport: SSE (typical for provider APIs) or WebSockets; a FastAPI layer proxies with StreamingResponse/generators. Buffering proxies (nginx defaults) silently un-stream you — a classic gotcha.
• Streaming + structured output conflict: you can't json.loads half a document. Options: stream to the UI but parse-on-complete for machine consumers; incremental/partial-JSON parsing; or stream only designated free-text fields.
• Downstream effects: mid-stream failures need re-request logic (you can't "resume" a completion); moderation on streams must be incremental; non-user-facing pipeline calls don't need streaming at all — knowing when it doesn't matter is also signal.

Token budgeting

Context windows and wallets are both finite; budget explicitly per request class:

budget = system prompt + tools + history + retrieved context + input + max_output

• Set max_tokens deliberately per call type — it caps cost, bounds worst-case latency, and (for agents) prevents a runaway generation from eating the loop's budget.
• Budget the inputs: cap retrieved-context size (top-k × chunk size), truncate/summarize history and tool outputs, keep tool definitions lean — in agent loops, tool results are usually the context hog (trim at the source).
• Count, don't estimate, where it matters: tokenizer libraries / count-tokens endpoints for pre-flight checks on user-supplied content.
• Per-tenant/per-task ceilings: an agent run gets a total token allowance; exceeding it is a handled failure (escalate/truncate), not a surprise invoice.

03Caching: exact, semantic, prefix

Three different mechanisms — being precise about which is which reads very well:

04Cost engineering

Cost = Σ over calls of (input_tokens × input_price + output_tokens × output_price), with output typically several× the input price. Levers in rough order of leverage:

1. Model routing — the biggest single lever: route each task to the cheapest model that passes evals (small model for triage/extraction, workhorse for drafting, frontier for hard reasoning). Order-of-magnitude price gaps between tiers mean routing 70% of traffic down-tier transforms the bill. Evals are the referee — routing without evals is just hoping.
2. Prompt & context diet — trim boilerplate, cap retrieved context, summarize history, compress tool outputs. Token reduction is a permanent annuity.
3. Prefix caching — restructure prompts for cacheability (§03).
4. Output discipline — concise-output instructions, structured formats, tight max_tokens (output is the expensive direction).
5. Batch APIs — ~50% off for async batch (results in minutes–hours). Perfect for non-interactive volume: nightly evals, backfills, bulk classification.
6. Fine-tune small to replace big — distill a frontier-prompted behavior into a cheap model once volume justifies the fixed cost.
7. Measure per unit of value — cost per resolved ticket / per fix PR / per tenant, and alert on anomalies. A looping agent shows up as a cost spike before anyone notices functionally; budget alarms are incident detection.

05Reliability: retries, fallbacks, circuit breakers

The dependency fails in more ways than a normal API: 429s (rate limits), 5xx/overloaded, timeouts, and semantic failures — malformed JSON, refusals, empty or off-contract output. Handle all of them:

• Retries with exponential backoff + jitter, honoring Retry-After; retry 429/5xx/timeouts; don't blind-retry 400s (deterministic failures). Distinguish transport retries (same request) from semantic retries (re-prompt with the validation error appended — "your output failed schema X, fix it" — which fixes a surprising fraction of malformed outputs). Cap both; then fall back or fail cleanly.
• Fallback chains across models/providers: primary model → same-provider sibling → second provider. Requires a gateway abstraction (yours!): one internal request/response schema, per-provider adapters normalizing message formats, tool-call formats, error taxonomies, token accounting.
• Circuit breakers: provider erroring above threshold → open the circuit, route around it for a cooldown, probe before restoring. Prevents retry storms against a struggling provider (which worsens their brownout and burns your latency budget).
• Timeouts everywhere, sized per call class (a reasoning call legitimately runs minutes; a router call should die in seconds); hedging (fire a second request if p95 is breached) for latency-critical paths if budget allows.
• Graceful degradation as product design: if all LLM paths fail, what happens? Queue-and-retry-later (fine for async pipelines like Sentinel's), serve a cached/template answer, or surface honest failure — decided per feature, in advance.

06Observability for LLM apps

Classic observability tells you the request was 200-in-3s. LLM observability must also answer: what did the model see, what did it say, what did it cost, and was it any good?

Tracing

Treat each user request / agent run as a trace; each LLM call, tool call, and retrieval as spans. Per LLM span, record: model + version, full rendered prompt (system + messages), parameters, completion, token counts (in/out/cached), latency (TTFT + total), finish reason, error/retry/fallback path taken. For agent runs the trace is the debugging artifact — "which tool result poisoned the context?" is answerable only with full transcripts. Tooling names to drop: LangSmith, Langfuse, Braintrust, Arize Phoenix, or OTel GenAI semantic conventions on your existing stack — the concept matters more than the vendor.

Metrics to dashboard

• Token throughput and cost — per feature, per tenant, per model
• Latency — TTFT and total, per model
• Error taxonomy — rate-limit vs timeout vs malformed-output vs refusal
• Cache hit rates (exact / semantic / prefix), fallback activation rate
• Agent-specific: loop iterations, tool-failure rate, tokens per completed task, human-gate approval/rejection rate

Online quality signal: you can't assert correctness per-request, but you can monitor proxies — schema-validation failure rate, refusal rate, output-length drift, judge-scored samples, user feedback/regeneration rate. Drift in these catches silent model updates and prompt regressions between eval runs.

07Evals — testing a probabilistic system

The core inversion: normal tests assert exact outputs; LLM outputs are distributions. So you test properties, scores, and rates over a dataset, with thresholds. "Do you eval, and how" is the question that separates production AI engineers from API tinkerers.

The eval pyramid: golden set → judge → human

Regression testing prompts (evals in CI)

Prompts, model versions, retrieval configs, and agent scaffolds are all behavior-defining artifacts → version them, and run the eval suite in CI on any change to them: "prompt change → eval run" exactly as "code change → test run". Also re-run on provider model updates — the dependency shifts under you silently. Tiering mirrors the test pyramid: fast cheap checks per-commit; full judged suite pre-merge/nightly; expensive end-to-end agent evals on schedule. Track score trends, not just pass/fail — slow degradation is the common failure shape.

A/B testing and shadow mode

Offline evals rank candidates; online A/B confirms on real traffic with product metrics (task completion, user acceptance/gate-approval rate, regeneration rate, downstream conversions). LLM specifics: high output variance → longer runs or paired/counterfactual designs; log the variant into traces so quality metrics segment cleanly; and for risky changes, shadow mode first — the new variant runs silently, outputs logged and judged, users see the old one. Shadow mode is the natural pattern for systems with human gates: run the new prompt through the gate pipeline and compare approval rates before switching.

08Safety basics: prompt injection

The vulnerability class every AI engineer must explain fluently in 2026. Root cause: the model has one input channel — the context — in which trusted instructions and untrusted data are concatenated as indistinguishable tokens. Any text the model reads can act like instructions. It's SQL-injection-shaped, but there is no reliable equivalent of parameterized queries — delimiters and "ignore instructions in the data" prompts lower, never eliminate, the success rate. Saying "unsolved; managed architecturally" is the correct expert position.

Mitigations, layered

Adjacent basics worth one line each: secrets/PII redaction in prompts and traces; output moderation for user-facing text; supply-chain trust for third-party MCP servers/tools (their descriptions and outputs enter your context); insecure-output-handling — never eval/render/execute model output without the same sanitization you'd give user input. The model is a user.

09Rapid-fire interview Q&A

10Self-test

← Agents, tools & MCP Interview talking points →