03 · System Design
Reliability & Observability
This is where you have the deepest lived experience relative to your level: you took an endpoint from 71 seconds to 2 seconds using distributed tracing, you run K8s deployments, and you keep a billing-grade pipeline alive. Most SDE-1/2 candidates know these topics from blog posts; you know them from pager reality.
The framing sentence. Open any reliability discussion with:
"Failures aren't exceptional in distributed systems — they're the steady state. So I design assuming every dependency will be slow or down sometimes, and the system's job is to degrade predictably instead of collapsing."
Everything on this page is a tool for that one goal.
§1SLIs, SLOs (and SLAs)
Indicator — a measurement. "p99 latency of POST /v1/chat", "% of 5xx responses", "Kafka consumer lag."
Objective — internal target on an SLI. "99.9% of requests succeed over 30 days"; "p99 < 500ms."
Agreement — the contractual, external version: SLO + penalties. Always looser than your SLO.
Error budgets — the idea that makes SLOs useful
99.9% over 30 days = 43.2 minutes of allowed badness per month. Budget remaining → ship fast. Budget burned → freeze features, fix reliability. It converts "be reliable" from vibes into an engineering currency that gates decisions.
Availability table worth memorizing:
99% → 7.3 h/month 99.9% → 43.8 min/month
99.95% → 21.9 min/month 99.99% → 4.4 min/monthChoosing SLIs (say this): measure what users experience, at the edge — request success rate and latency at the gateway, not CPU% on a pod. Use percentiles, never averages: an average hides that 1% of your requests take 30s. p99 matters because your busiest users hit it most often — a user making 100 calls has a 63% chance of hitting a p99 event.
Your hook: "With 50+ enterprise clients on the platform, latency and availability aren't abstract — enterprise contracts come with expectations. And LLM inference has fat-tailed latency by nature, so we track p95/p99 per route and per tenant, not averages."
§2Timeouts, Retries, Backoff + Jitter
Timeouts — the most underrated setting in distributed systems
No timeout = a hung dependency consumes your threads/connections until you are down too. Every network call gets a timeout, chosen from the dependency's actual p99 (e.g., p99 is 200ms → timeout 500ms), not a default 30s.
Timeout budgets: if the gateway gives a request 2s total, the service must give its DB call less than that, minus its own work — timeouts should shrink as you go deeper in the call chain, or the outer caller gives up while inner work continues uselessly.
Retries — powerful and dangerous
Retry helps only for transient failures (network blip, brief overload, 503). Never retry non-idempotent operations without an idempotency key (see building blocks §13). Never retry 4xx — your request is wrong, and it won't get righter.
The retry storm: dependency slows → everyone times out → everyone retries → 3× load on an already-drowning service → hard down. Retries convert partial failure into total failure unless bounded.
Exponential backoff + jitter
attempt 1: wait 100ms Without jitter, 1000 clients that
attempt 2: wait 200ms failed together retry TOGETHER —
attempt 3: wait 400ms synchronized waves that re-kill the
attempt 4: wait 800ms (cap ~10s) recovering service.
FULL JITTER: sleep = random(0, min(cap, base × 2^attempt))
→ spreads the wave into a smooth trickle.Complete answer in one breath: "Bounded retries (2–3), exponential backoff with full jitter, only on idempotent operations, with a retry budget so retries can't exceed ~10–20% of traffic, and ideally only at ONE layer of the stack — if the gateway, the service, AND the client all retry 3×, one failure becomes 27 attempts."
Your hook: "LLM providers rate-limit and hiccup routinely, so our gateway's retry/fallback logic is load-bearing — bounded retries with backoff, and provider fallback when a primary is degraded." (Bridges straight into circuit breakers.)
§3Circuit Breakers
Retries handle brief failures; circuit breakers handle sustained ones. Pattern: track a dependency's recent failure rate; past a threshold, stop calling it — fail fast instead of queueing doomed 30s timeouts.
Why fail-fast matters: waiting on a dead dependency holds threads, connections, and memory — the caller dies of resource exhaustion. That's how cascading failures propagate; the breaker is the firewall. On open circuit: serve a fallback (cache, default, queue-for-later), don't just error.
Your hook: "In an LLM gateway this is concrete: if a provider's error rate spikes, we trip that provider open and route to a fallback provider — users see slightly different model behavior instead of timeouts. The breaker state itself is a metric worth alerting on."
§4Bulkheads
Named after ship compartments: partition resources so one flooding compartment doesn't sink the ship. Separate connection pools / thread pools / pod pools per dependency or per tenant class — slow dependency A exhausts its pool of 20 connections, while B and C sail on.
Multi-tenant angle (yours): one tenant's burst shouldn't degrade everyone. Per-tenant rate limits ARE a bulkhead at the front door; per-tenant concurrency caps at the service; optionally isolated capacity for whale tenants.
Your hook: "With 50+ enterprise clients, tenant isolation is the difference between one noisy tenant and one angry tenant — versus fifty."
§5Graceful Degradation
Design the answer to "what breaks first, and what do users see?" A degraded-but-usable system beats a down system:
- Search ranking service down → serve recency-sorted results.
- Recommendations down → serve popular items.
- LLM provider down → fall back to a secondary provider/model; if all down → queue and notify, or return a clear, fast error.
- Redis down → fail open or fail closed? For rate limiting: usually fail open (allow traffic, log loudly) because blocking ALL paying customers over a limiter outage is the worse failure. Saying this tradeoff out loud is a strong signal.
Degradation must be designed and tested, not hoped for — name feature flags / kill switches as the mechanism.
§6Dead Letter Queues (DLQ)
In any queue consumer: some messages will never succeed (malformed payload, bug, poison pill). Infinite retry on a poison message halts the partition behind it — one bad message becomes an outage.
[Kafka topic] → [consumer: try N times w/ backoff]
│ still failing
v
[DLQ topic] → alert → human/tooling inspects,
fixes, REPLAYS to main topicRules: attempts + error metadata travel with the DLQ'd message; DLQ depth is an alerting metric (nonzero = a bug you haven't met yet); build the replay path before you need it. Distinguish transient (retry in place) vs permanent (DLQ immediately) failures.
Your hook: "In our Kafka→OpenSearch pipeline, a malformed event must not stall the partition behind it — park it, alert, keep the pipeline draining, replay after the fix."
§7Distributed Tracing — and Your 71s→2s Story
Metrics say something is slow; logs say what one process did; traces show where the time went across services. A trace = one request's tree of spans (operation + start + duration + service), linked by a trace ID propagated through every hop (W3C traceparent header; OpenTelemetry is the standard toolkit). The gateway is the natural place to start the trace and inject the ID.
TRACE: POST /process total: ██████████████████████ 71s
├─ gateway auth ▌ 40ms
├─ service A: handle ██████████████████████ 70.8s
│ ├─ db query (x N!) ████████████████████ 68s ← THE CULPRIT
│ │ └─ N+1: one query per item, serially
│ └─ downstream call █ 2s
└─ response serialize ▌ 100msYour story, told as a methodology — rehearse this ~90-second version:
"We had an endpoint taking 71 seconds. Instead of guessing, I went to the tracing data. Step one: get a full trace of a slow request and read the span waterfall — the question is always 'which span owns the time?' It wasn't spread evenly; the time concentrated in one service's data access. Step two: zoom into that span's children — the shape of the problem was visible in the trace structure: many small sequential operations instead of few batched ones. Step three: fix the access pattern — batch the work, eliminate the redundant round trips. Step four: verify with the SAME traces — the endpoint went to 2 seconds, a ~35× improvement, and the waterfall now showed a healthy profile. The meta-lesson I took: never optimize on a hunch; make the system tell you where the time goes. Tracing turned a mystery into a 30-minute read."
Why interviewers love this: it demonstrates the loop — observe → hypothesize → fix → verify with the same instrument. That's senior behavior at SDE-2 price.
Sampling (one-liner to volunteer): tracing every request is expensive at volume → head sampling (random N%) vs tail sampling (keep the slow/error traces — the interesting ones).
§8Metrics, Logs, Traces — The Three Pillars
| Pillar | Nature | Question answered | Cost profile |
|---|---|---|---|
| Metrics | Pre-aggregated numbers over time | "Is something wrong?" (alerting) | Cheap, fixed — but beware label cardinality |
| Logs | Discrete events, arbitrary detail | "What exactly happened here?" | Expensive at volume; structure them (JSON), index them |
| Traces | Per-request causal tree | "WHERE across services?" | Sampled |
Workflow to narrate: alert fires on a metric → traces localize the offending service/span → logs (jump via trace ID stamped in every log line) give the exact error. Correlation via a shared trace/request ID is what makes the three pillars one system.
Metrics vocabulary worth dropping: RED method for services (Rate, Errors, Duration) / USE for resources (Utilization, Saturation, Errors); Prometheus-style counters/gauges/histograms; percentiles from histograms.
Your hook: "Our observability doubles as a product: the same per-request metering that feeds dashboards also feeds billing — usage metering for 50+ clients is observability with an invoice attached, which means accuracy requirements are billing-grade, not best-effort. And log search itself is a system I run: logs/events flow through Kafka into OpenSearch — batched bulk ingestion — where 5,000+ concurrent users query them."
§9Health Checks
"Is the process alive?" Fail → K8s restarts the container. Keep it dumb (deadlock detection), or restarts amplify outages.
"Can I serve traffic NOW?" Fail → K8s removes the pod from Service endpoints, no restart. Checks deps: DB reachable, caches warm, config loaded.
Grace period for slow boots before liveness applies.
The classic mistake (name it): putting a hard dependency check in liveness — DB blips for 10s → every pod restarts simultaneously → thundering-herd reconnect → worse outage. Dependency checks belong in readiness. Also: a deep readiness check that calls a shared dependency from 200 pods every 5s is itself load — keep probes cheap.
Your hook: "Every one of our 20+ services on K8s defines liveness and readiness probes — readiness gating is also what makes rolling deploys safe: the new pod takes traffic only when it proves it's ready."
§10Deployments: Rolling, Blue-Green, Canary
(K8s default) Replace pods gradually (maxSurge/maxUnavailable), readiness-gated.
Pro: no extra infra, zero-downtime.
Con: two versions live simultaneously; slow full rollback.
Full parallel env; flip traffic at LB/gateway; keep old warm.
Pro: instant rollback (flip back); test green with prod parity.
Con: 2× infra cost; DB schema shared by both — the real coupling.
1–5% of traffic → new version; watch error rate/p99; ramp 5→25→100.
Pro: real-traffic validation, small blast radius.
Con: needs weighted routing (gateway!) + good metrics + patience.
Two things to always add:
- "Two versions run at once in every strategy — so DB migrations must be backward-compatible": expand → migrate → contract (add nullable column, dual-write/backfill, then drop old). Never a breaking migration in one deploy.
- Deploy ≠ release: feature flags decouple shipping code from exposing behavior — the cheapest rollback is a flag flip.
Your hook: "We do rolling deploys on K8s daily — readiness probes gate each step. And Kong is exactly where canary weighting lives: the gateway can split 95/5 by weight, which is the natural canary mechanism for our platform."
§11Backpressure
The problem: producer outruns consumer. Unbounded queues in between just delay and worsen the crash (memory bloat, then total loss). A system without backpressure doesn't degrade — it detonates.
Strategies, in escalation order:
- Bound every queue/pool — bounded queue + rejection beats unbounded queue + OOM.
- Slow the producer: TCP does this natively; app-level = blocking writes, semaphores, windowing (Kafka consumers pull at their own pace — pull-based consumption is built-in backpressure, a key reason the Kafka-in-the-middle architecture works).
- Shed load: return
429/503 + Retry-Afterearly at the edge (cheap) rather than timing out late in the stack (expensive). Rate limiting IS pre-emptive load shedding — you built this. - Degrade: sample, batch harder, drop low-priority work first (analytics before billing).
producer 10K/s ──> [bounded buffer] ──> consumer 5K/s
│ full
├─ block producer (backpressure)
├─ reject new (shed load, 429)
└─ drop by priority (degrade)
Kafka's answer: the LOG absorbs the burst durably;
consumer lag grows and is VISIBLE; you scale consumers to drain.Your hook: "Our pipeline handles this at two points: Kafka absorbs bursts durably so OpenSearch is never force-fed — consumers pull and bulk-index at the rate the cluster sustains, and lag is the pressure gauge. At the front door, per-tenant rate limits on Kong shed excess load before it costs us anything downstream."
§12Rapid-Fire Q&A Prep
"How do you know your service is healthy?"
SLIs at the edge: success rate, p95/p99 latency, plus consumer lag for pipelines. Alert on SLO burn rate, not on CPU.
"A downstream is slow. Walk me through what happens."
Timeout bounds the damage → bounded retries w/ jitter → breaker trips → fallback/degrade → bulkhead contains it → alert fires → trace localizes it. (This one answer tours the whole page.)
"How do you deploy without downtime?"
Rolling with readiness gates; canary via gateway weights for risky changes; expand-migrate-contract for schema; feature flags for instant rollback.
"Message keeps failing in your pipeline?"
Classify transient vs permanent; bounded retry with backoff; DLQ + alert; replay after fix; idempotent consumers make replay safe.
"Tell me about a hard performance bug."
The 71s→2s tracing story (§7). Have the 90-second version rehearsed: waterfall → culprit span → N+1 shape → batch the access pattern → verify with the same traces → ~35× improvement.
§13Self-Test
What's the difference between an SLI, an SLO, and an SLA?
SLI = the measurement (p99 latency, success rate). SLO = internal target on that SLI (99.9%/30d). SLA = the contractual external version with penalties, always looser than the SLO.
Your SLO is 99.9% over 30 days. How much downtime is that, and what's an error budget for?
~43 minutes/month. The budget is engineering currency: remaining budget → ship features fast; burned budget → freeze and fix reliability. It gates decisions instead of "be reliable" vibes.
Why must timeouts shrink deeper in the call chain?
Timeout budgets: if the gateway allots 2s total, inner calls must fit inside that minus local work — otherwise the outer caller gives up while inner work continues uselessly, burning resources for answers nobody is waiting on.
Recite the one-breath retry answer.
Bounded retries (2–3), exponential backoff with full jitter, only on idempotent operations, with a retry budget (~10–20% of traffic), and only at ONE layer of the stack — three layers each retrying 3× turns one failure into 27 attempts.
Draw the circuit breaker state machine from memory.
CLOSED (counting failures) → threshold crossed → OPEN (fail fast, no network) → cooldown elapses → HALF-OPEN (few probes) → probes succeed → CLOSED; probes fail → back to OPEN. On open: serve a fallback, don't just error.
Fail open or fail closed when Redis (your rate limiter) dies?
Usually fail open for rate limiting: allow traffic and log loudly, because blocking ALL paying customers over a limiter outage is the worse failure. Naming this tradeoff out loud is the signal.
Why does a DLQ exist, and what are the three rules?
Poison messages never succeed; infinite retry stalls the partition — one bad message becomes an outage. Rules: carry attempts + error metadata with the message; alert on DLQ depth (nonzero = unmet bug); build the replay path before you need it.
What goes in a liveness probe vs a readiness probe — and the classic mistake?
Liveness: dumb "process alive" checks only (fail = restart). Readiness: dependency checks — DB reachable, caches warm (fail = pulled from endpoints, no restart). Mistake: DB check in liveness → a 10s DB blip restarts every pod at once → thundering-herd reconnect.
Why must DB migrations be backward-compatible in every deploy strategy?
Two versions run at once in all of rolling/blue-green/canary. So: expand → migrate → contract (add nullable column, dual-write/backfill, drop old later). Never a breaking migration in one deploy. And deploy ≠ release — feature flags are the cheapest rollback.
Name the four backpressure strategies in escalation order.
1) Bound every queue/pool. 2) Slow the producer (TCP natively; Kafka pull-based consumption is built-in backpressure). 3) Shed load early at the edge (429/503 + Retry-After). 4) Degrade — drop low-priority work first (analytics before billing).
Tell the 71s→2s story in 90 seconds.
Trace the slow request → read the span waterfall ("which span owns the time?") → time concentrated in one service's data access → children show many small sequential ops (N+1) → batch the access pattern → verify with the SAME traces → 2s, ~35×. Meta-lesson: never optimize on a hunch; make the system tell you where the time goes.