03 · System Design

Designing LLM-era systems (your moat)

Most candidates at your level have never operated an LLM system in production. You run one at 2M+ requests/month for 50+ enterprise clients. When the question is open-ended, steer here — these four designs are your strongest territory, and the first one is literally your job.

§0Steer the interview onto your turf

Interviewers routinely leave the door open: "design an API platform", "design something interesting", "design a system you know well". Walk through that door deliberately. The universal line:

Say it: "Can I take this in the direction of LLM serving? It's what I run in production, so I can go unusually deep."

If the interviewer asks…	Pivot to	Your opening line
"Design an API gateway / API platform"	(a) LLM gateway	"This is essentially my day job — let me design the gateway I actually run, from scratch."
"Design a billing / usage-metering system"	(a), billing deep-dive	"The hardest metering I know is token billing for LLMs — can I use that as the concrete case?"
"Design something interesting / a system you know well"	(c) Sentinel	"I built an agentic workflow system in production — Sentinel. Let me design that class of system."
"Design search / 'chat with your docs' / a Q&A product"	(b) RAG pipeline	"This is really a search system with an LLM at the end — and I run production search infrastructure (OpenSearch)."
"Design Zapier / a notification or automation platform"	(d) Webhook platform	"I consume Stripe/Razorpay webhooks in production and run the queue-worker architecture this needs — let me build the platform side."
Anything vague or open-ended	whichever fits	"Can I take this in the direction of LLM serving? It's what I run in production, so I can go unusually deep."

(a)LLM API Gateway / Proxy

Framing — say this first: "This is essentially my day job — I run Kong on Kubernetes fronting an LLM-serving platform: auth, per-tenant rate limiting, and billing for 50+ enterprise clients at 2M+ requests/month. Let me design it as if from scratch."

Requirements

Functional: one API for chat/completions across multiple upstream providers; per-tenant API keys + auth; per-tenant rate limits and quotas; token-based usage metering → billing; streaming responses (SSE); provider fallback/routing; per-tenant config (allowed models, spend caps). Out of scope: training/hosting the models.

Non-functional: gateway overhead ≤ a few ms (the LLM call itself is 1–30 s — don't be the slow part); billing metering must be accurate and never lost (money); high availability (every tenant's traffic flows through you); tenant isolation (one tenant's burst can't starve others).

Estimation — the insightful inversion

2M req/month ≈ 0.8 QPS avg, ~5-10 peak — compute is NOT the problem.
What IS: · long-lived connections (streams held open 5-60s
           → connection slots, not CPU, are the scarce resource)
         · per-request COST (an LLM call can cost real money —
           metering errors compound directly into revenue errors)
         · provider rate limits & outages (external, uncontrollable)
Billing events: 2M/month — trivial volume, but LOSS-INTOLERANT.

Make the observation out loud: "My QPS is small but per-request cost and duration are huge" — this inversion is itself an insight few candidates can offer, because it flips which resources are scarce (connection slots and money, not CPU).

API

POST /v1/chat/completions      (OpenAI-compatible shape — industry lingua franca)
  headers: Authorization: Bearer <tenant_api_key>
  body: { model, messages[], stream: true|false, max_tokens, ... }
  stream: SSE — "data: {delta}\n\n" chunks, terminated by "data: [DONE]"
GET  /v1/usage?from=&to=       → per-model token counts, cost (tenant-facing)
Admin: CRUD tenants, keys, limits, model allowlists, spend caps

High-level design

 50+ tenants
     │
     v
 [ KONG / gateway layer (N replicas on K8s) ]
 │ 1. authn: API key → tenant (Redis-cached key lookup)
 │ 2. rate limit: per-tenant token bucket (Redis, atomic Lua)
 │ 3. spend cap check: tenant balance/quota (cached, async-refreshed)
 │ 4. route → proxy service
 v
 [ LLM Proxy service ]
 │ · model → provider routing table (config as data)
 │ · provider adapters (normalize request/response shapes)
 │ · circuit breaker per provider; fallback chain: primary → secondary
 │ · SSE pass-through streaming (no buffering!)
 │ · count tokens: prompt (pre) + completion (from stream/usage field)
 ├──────────────> [ Provider A ]  [ Provider B ]  [ self-hosted models ]
 │
 └─ after response/stream end:
    emit usage event ──> [ Kafka: "usage_events" ] ──> [ Billing aggregator ]
       { event_id, tenant, model,                        │ idempotent upsert
         prompt_tokens, completion_tokens,               v
         latency_ms, status, ts }              [ Postgres: usage ledger ]
                                                 → invoices (Stripe/Razorpay)

Provider Fallback Stepper

Pick a failure scenario and step the request through the gateway. Watch where the retries, the circuit breaker, and the degradation decisions happen.

Client
tenant app, SSE consumer

Gateway
auth · rate limit · spend cap · metering

Provider A
primary in the fallback chain

Provider B
secondary / fallback

Cache / Queue
graceful-degradation path

Pick a scenario above, then hit Step to follow the request.

Deep dive 1 — token billing that never loses money

Request-count billing is wrong for LLMs: a 10-token and a 100K-token request differ ~10,000× in cost. Meter tokens per model (each model has its own price). The durability chain, link by link:

1 · Emit server-side, at stream end. Count what was generated, not what was delivered — if the client disconnects mid-stream you already paid the provider for those tokens. Never rely on the client to report usage.

2 · Usage event → Kafka, durable and acked. The request path never waits on the ledger; the event outlives any process crash.

3 · Billing aggregator consumes at-least-once. Duplicates are allowed to happen here — the next link makes them harmless.

4 · Idempotent ledger writes — unique event_id, upsert into Postgres. At-least-once delivery + idempotent aggregation = exactly-once billing effect. Losing an event = giving away money; double-counting = an angry enterprise client. Both unacceptable — hence this pipeline instead of fire-and-forget metrics.

5 · Ledger → invoices (Stripe/Razorpay), plus a reconciliation job: sum of ledger vs provider invoices, alert on drift. That's how you test billing accuracy, not hope for it.

Spend caps — two consumers of one stream. Enforcement wants real-time-ish counters (Redis, incremented as events flow) even though the ledger is async: approximate-fast for enforcement, exact-slow for invoicing. And if Kafka is down: local durable disk spool at the proxy + replay; degrade to sync-write-to-ledger if the buffer fills; never silently drop usage.

Deep dive 2 — SSE streaming quirks

Streaming through a gateway — the traps:
· SSE = long-lived HTTP; the gateway must not buffer (disable proxy buffering — the classic misconfiguration that turns a stream into one big blob at the end).
· Timeouts must be idle timeouts (no chunk for N seconds), not total-duration — legitimate streams run 5–60 s.
· Capacity-plan connection slots, not QPS: thousands of open streams ≠ thousands of requests/second.
· Least-connections load balancing beats round-robin when request durations vary 100×.
· Graceful deploys must drain long-lived streams: K8s terminationGracePeriodSeconds + preStop.

Deep dive 3 — provider fallback & routing (what the stepper showed)

Per-provider circuit breaker (error-rate/latency threshold trips it) + a fallback chain. Subtleties worth naming out loud: the fallback model may differ subtly (tenant opt-in / per-tenant allowlist); retry-after-partial-stream is ugly — if tokens already streamed to the client you can't transparently switch mid-response, so fallback applies to pre-first-token failures; on provider 429s, respect Retry-After and spread across keys/accounts where applicable. Keep routing as data: a model → [provider chain] table, hot-reloaded — which also enables cost-based routing (cheapest healthy provider) as an extension.

(b)RAG Pipeline

Framing: "Retrieval-augmented generation — the standard architecture for 'chat with your documents'. Two halves: an ingestion pipeline (offline-ish) and a query path (online). It's really a search system with an LLM at the end — and I run production search infrastructure (OpenSearch), so I'll lean on that."

Functional: ingest documents (PDF, HTML, wiki, tickets); users ask questions; answers grounded in retrieved sources with citations; per-tenant corpus isolation; incremental updates. Non-functional: query p95 within a few seconds (retrieval <200–500 ms; the LLM call dominates); freshness in minutes, not days; retrieval quality is THE product metric — needs evaluation, not vibes; tenant isolation is a security boundary — retrieval must NEVER cross tenants.

Corpus: 1M documents × ~5 chunks avg = 5M chunks.
Embeddings: 5M × 1536 dims × 4 B ≈ 30 GB of vectors — one beefy vector
index node or a small cluster; this is NOT big data (a common misread).
Query: 100 concurrent users → tens of QPS retrieval — modest.
Ingestion burst: the initial 1M-doc backfill is the heavy day-one load
→ queue-driven workers (embedding API rate limits are the bottleneck).

 INGESTION (async, queue-driven — this is an ETL pipeline)
 [Sources: uploads, connectors, crawlers]
        │ doc events
        v
 [ Kafka: "docs" ] ─> [ Ingestion workers ]
                       │ 1. parse/extract text  2. CHUNK
                       │ 3. embed chunks (batched)  4. upsert by chunk_id:
                       ├──> [ Vector index ]         (embeddings)
                       ├──> [ Keyword index (BM25) ] (OpenSearch)
                       └──> [ Metadata store (PG): docs, chunks, ACLs ]

 QUERY PATH (online)
 user query → [ Query service ]
   │ 1. embed the query
   │ 2. HYBRID retrieve (parallel): vector top-50 + BM25 top-50 → merge (RRF)
   │ 3. filter: tenant_id + ACLs (in the index query, NOT post-hoc!)
   │ 4. RERANK top-50 → top-5 (cross-encoder)
   │ 5. build prompt: system + top-5 chunks w/ source tags + question
   │ 6. LLM call — through the gateway from design (a)! — stream answer
   v
 answer + citations [chunk → doc → link]

Deep dives

Chunking (looks trivial, decides quality). Too big → retrieval imprecise, wastes context tokens; too small → fragments lack context. Practical default: ~300–800 tokens, 10–20% overlap, split on structural boundaries (headings, paragraphs), not blind character counts. Enrich each chunk with metadata (title, section path, URL, updated_at, tenant_id, ACL tags) — that powers filtering AND citations. On doc update: re-chunk with deterministic IDs (doc_id + position + content_hash) → upsert changed, delete orphans. Idempotent by construction — your OpenSearch upsert discipline.

Hybrid retrieval + reranking. Vector search captures semantics ("reset my password" ≈ "credential recovery") but is weak on exact tokens — error codes, product names, IDs. BM25 is the mirror image. Run both, merge with Reciprocal Rank Fusion (rank-based, no score-calibration headaches). Then rerank: first-stage retrieval optimizes recall (50 candidates, cheap); a cross-encoder reads (query, chunk) pairs together for precision on the top handful. Two-stage retrieval = the classic search architecture: cheap recall → expensive precision. Index choice: HNSW-based — pgvector at small scale (vectors next to metadata); "I'd likely use OpenSearch k-NN since I already operate OpenSearch — one system for both BM25 and vectors."

Evaluation (separates you from tutorial-followers). "You can't improve what you don't measure. I'd build a golden set of (question → relevant chunks / reference answer) pairs and track retrieval recall@k and answer faithfulness/groundedness — checkable with an LLM-as-judge, spot-audited by humans." Run the eval suite on every pipeline change (chunk size, embedding model, k, reranker) — treat retrieval config changes like deploys: measure, canary, roll back. Log which chunks fed each answer.

(c)Agentic Workflow System — your Sentinel

Framing: "I built Sentinel — an agentic workflow system (in production it generates 100+ fix PRs/month behind an approval gate, and cut incident-resolution effort ~70%). The core design problem: an LLM agent that takes real actions is a side-effect machine driven by a non-deterministic planner, so the architecture is about containment: durable task state, sandboxed tool execution, human approval gates for risky actions, and a complete audit trail."

Functional: accept a task; agent loop = LLM plans → picks tool → executes → observes → repeats; tool catalog; human approval gates for risky actions; full audit trail; resumability (task survives worker crash — including waiting hours for an approval); concurrent tasks. Non-functional: durable task state; safety > speed (an agent doing the wrong irreversible thing is the worst failure); isolation (tools can run untrusted/generated code); per-step observability; cost caps per task (agent loops can run away).

 user/API ──> [ Task API ] ──> [ Postgres: tasks, steps, approvals, audit ]
                  │                       ^ every state transition recorded
                  v                       │
          [ Queue: "tasks" ] ──> [ Orchestrator workers (N) ]
                                   │ AGENT LOOP (per step):
                                   │ 1. load task state + history from DB
                                   │ 2. LLM call: next action?  — through
                                   │    the LLM gateway (a)!
                                   │ 3. needs approval?
                                   │    ├─ yes → write approval_request,
                                   │    │   PARK task (state=WAITING),
                                   │    │   notify approver … resume on event
                                   │    └─ no ──┐
                                   │ 4. execute ▼ in SANDBOX
                                   │    [ isolated container: no ambient
                                   │      creds, egress allowlist, limits ]
                                   │ 5. persist step result (audit) → loop
                                   │ guards: max_steps, cost budget, deadline
                                   v
                            [ Audit log: append-only ]

Deep dives

Durable state machine, not an in-memory loop. Naive agents hold the loop in process memory → crash = lost task, or worse, repeated side effects. Instead: each step is a transaction — load state, decide, execute, persist result, advance. The worker is stateless between steps; any worker can pick up any task — your K8s cattle principle applied to agents. Crash between execute and persist → on recovery you don't know if the tool ran ⇒ tools must be idempotent or checkable (execution token per step, tool-side dedup) — the payments-timeout problem wearing a new hat. Say that connection out loud. Approval parking: WAITING_APPROVAL is just a state; the task burns zero compute for hours — durable state machines make "wait for a human" free.

Sandbox + permission model. Generated code is untrusted by definition: isolated container, no secrets in env, scoped short-lived credentials injected per call, egress allowlist, resource + wall-clock limits, workspace torn down after. Tool catalog entries declare a schema (validate LLM output before execution), a permission tier (auto / needs-approval / forbidden), rate limits, and cost — risk policy is data, per-tenant configurable, not code. Approval UX: the approver sees the exact proposed action + arguments + the agent's reasoning — approve / deny / edit-then-approve; a deny feeds back into the loop as an observation so the agent re-plans.

Audit trail & runaway protection. Append-only log of every prompt, response, tool call, result, and approval decision (who/when/what), with token/cost per step. Non-negotiable: debugging non-determinism (the trace IS the explanation), compliance ("why did the agent email that customer?"), and eval/replay (rerun history against a new model offline). Runaway guards: hard max_steps, per-task cost budget enforced via the gateway's metering (design (a) again — the pieces compose), loop detection (same tool + same args N times → halt and escalate).

(d)Webhook-Driven Automation Platform (Zapier-lite)

Framing: "A platform where external events trigger user-defined workflows — 'when X happens in service A, do Y in service B.' It composes everything: webhook ingestion, queues, per-tenant execution, retries, idempotency — and it's adjacent to what I operate: I consume Stripe/Razorpay webhooks in production and run the queue-worker architecture this needs."

Functional: per-tenant webhook ingest URLs; trigger → workflow of steps (filters, transforms, actions on third-party APIs); connectors (Slack, email, CRM, HTTP); per-execution run history users can inspect; retries. Non-functional: never lose an inbound event (ingest availability + durability IS the product); process at-least-once, effect at-most-once; tenant isolation (one tenant's 1M-event burst or infinite-loop workflow can't starve others); graceful handling of third-party rate limits; spiky load by nature.

10K tenants × avg 1K events/day = 10M events/day ≈ 120/s avg — but bursty
(one tenant's upstream can dump 50K events in a minute).
⇒ ingest tier does ALMOST NOTHING synchronously: validate → enqueue → 202.
Execution: avg workflow 3 steps, each an external API call (100ms-2s)
⇒ concurrency-bound, not CPU-bound → async workers, per-tenant fairness.

 3rd-party services
      │ POST https://hooks.platform.com/t/{tenant}/{hook_id}
      v
 [ Ingest tier (thin!) ]                 ← THE availability-critical part
 │ verify HMAC signature (per-hook secret)
 │ assign event_id, dedupe (Redis, provider event id)
 │ write to [ Kafka: "events", partitioned by tenant_id ]
 │ respond 202 in <50ms — NO workflow logic here
 v
 [ Workflow matcher ] → which workflows subscribe to this hook/event?
      │ enqueue one execution per matched workflow
      v
 [ Kafka: "executions" ] ──> [ Step executor workers ]
                              │ per-step: transform → call connector
                              │ connector rate limits: per-(tenant,connector)
                              │ step fails: retry w/ backoff → after N:
                              │   mark failed, DLQ, notify user
                              v
                        [ Run history store (PG): executions, steps,
                          inputs/outputs (truncated), status ]

Deep dives

Ingest tier as an availability fortress. A missed webhook is unrecoverable unless the sender retries — so this is the part that must never be down. Keep it dumb (verify, dedupe, enqueue, 202), overprovisioned, and independent of the execution tier's health: executors can be fully down and you lose nothing, lag just grows. Separating "accept" from "process" is the whole architecture. Signature verification per hook secret + timestamp tolerance (replay protection) — exactly what you do as a Stripe/Razorpay consumer, now from the platform side.

Per-tenant fairness. One hot tenant floods the shared execution topic → everyone lags. Options: partition by tenant (isolates ordering, but a hot partition remains), per-tenant concurrency caps in workers (simple, effective), or weighted fair scheduling from per-tenant queues (heavier). Start with concurrency caps + rate limits — the bulkhead/noisy-neighbor conversation from your multi-tenant day job.

Loops and runaway workflows. Workflow A's action triggers a webhook that triggers workflow A… Detect via execution-chain depth (propagate a chain ID + hop count, cap it), per-workflow execution rate limits, cost/step budgets — the same runaway-containment thinking as the agent system (c).

Exactly-once effects. Provider retries + your retries = duplicates everywhere ⇒ dedupe at ingest (provider event ID) AND idempotency keys on connector calls where the target API supports them; where it doesn't: best-effort dedup, and at-least-once documented honestly to users — interviewers respect the honesty about the limits.

§5The meta-move

All four designs share a skeleton: gateway (auth/limits/metering) → durable queue → idempotent workers → ledger/audit trail — and that skeleton is your production system. Whatever gets asked, you can usually walk to one of these four and then answer follow-ups from lived experience instead of theory. That is exactly the impression an SDE-2 loop is trying to detect.

§6What interviewers probe — test yourself

(a) "Why not just call the LLM providers directly from each service or client?"

Cross-cutting concerns handled once — auth, rate limits, metering, fallback — plus provider abstraction and central spend control. It's the classic API-gateway argument with money attached: without the gateway, every service re-implements billing-grade metering, and any one of them getting it wrong is a revenue bug.

(a) "What if Kafka is down? Do you drop usage events?"

Never silently drop. Local durable buffer (disk spool) at the proxy + replay when Kafka returns; degrade to synchronous write-to-ledger if the buffer fills. Losing a usage event is giving away money — availability of the billing pipeline degrades gracefully, durability never does.

(a) "How do you test billing accuracy?"

A reconciliation job: sum of the internal usage ledger vs the providers' invoices, alert on drift. Also rate-limit on two independent dimensions — requests/min AND tokens/min per tenant — so neither dimension can be gamed by the other.

(a) "A client disconnects mid-stream. Who pays for the tokens?"

You already paid the provider for every generated token, so you meter server-side on stream termination — count what was generated, not what was delivered — and never rely on the client to report usage. This is the edge case interviewers love; volunteering it signals production experience.

(b) "How do you guarantee tenant A never sees tenant B's documents?"

Filter inside the index query — a mandatory tenant_id clause enforced at the query-builder layer — never post-hoc filtering after retrieval. Then test it adversarially. Tenant isolation in RAG is a security boundary, not a relevance feature.

(b) "Retrieval returns nothing relevant. What does the user see?"

Threshold the retrieval scores and return an honest "I don't know" with an escalation path — hallucinated confidence is worse than no answer. Surface it as a product decision, not just an engineering one; that framing scores points.

Durable per-step state machine: every step persisted, so any worker resumes from the last persisted step. The dangerous window is crash-between-execute-and-persist — you don't know if the tool ran — so tools must be idempotent or checkable (execution token per step, tool-side dedup). Same shape as the payments-timeout problem.

Defense in depth — list all four layers: (1) schema validation rejects malformed calls before execution; (2) permission tier per tool (auto / needs-approval / forbidden); (3) human approval gate at the irreversibility boundary; (4) sandbox blast-radius containment — no ambient creds, egress allowlist, resource limits.

Lease/lock on the task — row lock or visibility-timeout semantics — and the steps are idempotent anyway, so even a lease race can't double-apply side effects. Belt and suspenders, and say both.

(d) "Your executor fleet dies for an hour. What do users experience?"

Zero event loss — everything is log-buffered in Kafka behind the thin ingest tier. Lag alarm fires, you scale out and drain; users see delayed runs, never lost ones. This answer only works because "accept" was separated from "process" — say that explicitly.

(d) "Step 3 of a 5-step workflow fails forever. Then what?"

Step-level retry policy with backoff → after N attempts, the execution is marked partial-failed, the event goes to a DLQ, the user is notified, and run history enables manual re-run from the failed step. Observability as a product feature — users debug via per-step inputs/outputs.

← Classic designs Your systems as answers →