03 · System Design — file 02
Building blocks
Every system design answer is assembled from ~15 LEGO bricks — and you operate most of them in production already. For each block: what it is from first principles, how it works, the tradeoffs, and where it lives in your stack so you answer from experience instead of from a blog post.
01Load balancers
First principles: one machine has a ceiling — CPU, memory, open connections. Past that ceiling you need many machines, and then something must decide which machine gets each request. That something is the load balancer.
+--------------+
clients --> | LB | --health checks--> pool
+--------------+
/ | \
[srv A] [srv B] [srv C] srv C failing health check
→ removed from rotationL4 — transport
Routes on IP:port, never reads the request. Fast, dumb. Example: AWS NLB, Kubernetes Services.
L7 — application
Reads HTTP — routes by path, header, cookie; terminates TLS, retries, rewrites. Example: ALB, Nginx, Envoy, Kong.
| Algorithm | How | When |
|---|---|---|
| Round robin | Next server in rotation | Uniform servers, uniform requests |
| Weighted RR | More traffic to bigger boxes | Mixed instance sizes; canary (95/5) |
| Least connections | Server with fewest open conns | Long-lived / variable-duration requests — crucial for LLM streaming, where one request can hold a connection 30s+ |
| IP hash / consistent hash | Same client → same server | Session affinity, cache locality |
Health checks: active (LB probes /healthz every N seconds; 3 consecutive fails → eject) vs passive (LB observes real request failures). Design point worth saying out loud: a health check should verify readiness (can I serve? is the DB reachable?) not just liveness (process up) — Kubernetes makes this distinction explicit with liveness vs readiness probes.
YOUR HOOK: "Kong is our L7 layer — it terminates client traffic for the whole platform, and Kubernetes Services do L4 balancing across pods behind it. For LLM traffic specifically, least-connections thinking matters because streaming responses hold connections open far longer than typical REST calls."
02Horizontal vs vertical scaling
Vertical — scale up
Bigger machine. Zero code changes, but a hard ceiling, a single point of failure, and cost grows super-linearly.
Horizontal — scale out
More machines. Near-unlimited and fault-tolerant, but it forces you to solve state: any pod must be able to serve any request.
The rule you say in interviews: "Scale the stateless tier horizontally; scale the stateful tier vertically first, then replicate, then shard — in that order, because each step adds an order of magnitude of operational complexity."
YOUR HOOK: "Our 20+ microservices scale horizontally via Kubernetes HPA — it's just a replica count. The databases are where scaling actually gets hard."
03Stateless services
A service is stateless when no request depends on server-local memory from a previous request. Sessions → JWT (client-side) or Redis (shared store); files → object storage; counters → Redis/DB.
Why it matters: statelessness is the precondition for horizontal scaling, rolling deploys, and self-healing — K8s can kill and reschedule any pod at any time.
YOUR HOOK: "Everything stateful in our platform is pushed to Postgres, Redis, or Kafka precisely so K8s can treat pods as cattle — reschedule, autoscale, rolling-deploy without draining sessions."
04Database replication
First principles: one DB node gives you a durability risk (disk dies → data gone), an availability risk (node down → app down), and a read ceiling. Replication attacks all three: keep copies on multiple nodes.
Leader–follower — the default answer
writes replication log (WAL)
app ────────────────> [ LEADER ] ══════════════════> [ FOLLOWER 1 ]
reads ─┐ ║ [ FOLLOWER 2 ]
└────────> (can read leader or followers)All writes go to the leader; the leader streams its write-ahead log to followers; reads can be served by followers (read scaling).
Sync vs async replication — the core tradeoff
| Synchronous | Asynchronous | |
|---|---|---|
| Leader acks write when | ≥1 follower confirmed | Immediately (local commit) |
| Write latency | Higher (network RTT) | Lowest |
| Data loss on leader crash | None (on sync follower) | Possible (unreplicated tail) |
| If follower is slow/down | Writes stall! | No impact |
Real systems often use semi-synchronous: one sync follower for durability, rest async. Say that.
Replication lag is the interview follow-up: async followers are seconds behind → a user writes, then reads a stale follower and doesn't see their own write. Fixes: read-your-own-writes (route that user's reads to the leader briefly), sticky routing, or monotonic read guarantees.
Failover — never a throwaway line. Leader dies → detect (heartbeat timeout), promote the most-caught-up follower, repoint clients. Dangers to name: split brain (old leader comes back thinking it's still leader → two leaders accepting writes; prevented by fencing/quorum) and lost writes (async tail that never replicated). "Failover is automatic" is a favorite interviewer probe.
YOUR HOOK: "Our Postgres runs primary + replicas; billing reads that must be fresh hit the primary, dashboard/analytics reads go to replicas where staleness is fine."
05Sharding (partitioning)
Replication copies ALL data to each node — it scales reads, not writes or storage. When the write rate or dataset outgrows one leader, you split the data: each shard holds a subset.
| Strategy | How | Pro | Con |
|---|---|---|---|
| Range | Keys A–M → shard 1, N–Z → shard 2 | Range queries work | Hotspots — everyone writing "today's" data hits one shard |
| Hash | shard = hash(key) mod N | Even spread | Range queries die; mod-N resharding moves ~everything |
| Directory | Lookup service maps key → shard | Max flexibility | The lookup service is itself a scaling/availability problem |
Choosing a shard key: you want (a) even distribution, (b) most queries answerable within one shard. For a multi-tenant platform, tenant_id is natural — all a tenant's queries stay on one shard — but a whale tenant becomes a hotspot. Mitigations: split the whale across sub-shards, or salt the key (tenant_id + hash(user_id) % 8).
Hotspots: the celebrity problem — one key gets 100x traffic. Mitigations: cache the hot key, split/salt it, or dedicate capacity.
Resharding pain (say this — it shows scar tissue): with mod N, going from 4→5 shards remaps ~80% of keys — a massive data migration under live traffic. Cross-shard queries need scatter-gather; cross-shard transactions need 2PC or (realistically) sagas/redesign. This is exactly why the next block exists.
06Consistent hashing
Problem it solves: hash(key) mod N remaps nearly all keys when N changes. Consistent hashing remaps only ~1/N of keys.
How: map the hash space onto a ring (0 to 2³²−1). Place each node on the ring (by hashing its ID). A key belongs to the first node clockwise from the key's hash. Remove a node → only its keys move, to the next node clockwise; every other key stays put.
The contrast to hammer: naive hash(key) mod N would remap ~all 12 keys the moment N changes. Consistent hashing moves only the keys in the affected arc — ~1/N of them.
Virtual nodes: with few physical nodes, ring gaps are uneven → each physical node is placed at 100–200 points on the ring ("vnodes"), smoothing distribution and letting a dead node's load spread across all survivors instead of dumping on one neighbor.
Where it's used (name-drop accurately): DynamoDB/Cassandra partitioning; Kafka's default partitioner is plain hash-mod (partition count is fixed, so that's OK); Redis Cluster uses hash slots (16384 slots — same idea, discretized); CDN/cache server selection; load balancer consistent-hash mode.
07Caching layers
First principles: the latency ladder — memory ~100ns, SSD ~100µs, cross-network DB query ~1–5ms, LLM call ~seconds. Caching = keep hot data on a faster rung. It's justified by skew: real workloads are Zipfian — a tiny fraction of keys gets most of the traffic.
Layers (client → origin): browser cache → CDN → API gateway cache → application cache (Redis) → DB internal caches.
Cache-aside (lazy)
The default. App: read cache → miss → read DB → populate cache with TTL. Simple; first request slow; stale until TTL/invalidation.
Write-through
Write cache + DB together. Fresh reads, slower writes.
Write-behind
Write cache, flush to DB async. Fast, risky — a loss window if the cache dies before flushing.
Eviction: LRU (the default answer), LFU, TTL. In Redis: allkeys-lru and friends.
The three classic failure modes (interviewers LOVE these):
1. Stampede / thundering herd: hot key expires → 10K requests hit the DB simultaneously. Fixes: per-key mutex ("single flight" — one request rebuilds, the rest wait), stale-while-revalidate, jittered TTLs.
2. Invalidation: "two hard problems in CS." Practical answers: short TTLs for tolerable staleness; explicit delete-on-write (DELETE cache_key in the write path); event-driven invalidation (CDC → invalidate).
3. Hot key: one key too hot for a single Redis node → replicate that key across nodes with client-side randomization, or promote it to an in-process cache.
YOUR HOOK: "We use Redis heavily — rate-limit counters, session/auth data, hot config. Rate-limit counters are a nice case study: they're write-heavy, tolerate small inaccuracy, and must be FAST, so Redis is the right home rather than Postgres."
08CDN
Cache static (and increasingly, dynamic) content at edge locations near users. Physics: India↔US RTT is ~200ms+; edge RTT is ~10–30ms.
- Pull CDN (the default): edge fetches from origin on first miss, caches per
Cache-Controlheaders. - Push CDN: you upload to the CDN — for large, predictable assets like video.
- Details worth one sentence each: cache keys (URL + headers), TTLs + purge/invalidation APIs,
stale-while-revalidate, signed URLs for private content. - In LLM/API systems the CDN matters less for JSON APIs (dynamic, personalized) but still terminates TLS close to users and absorbs DDoS.
09Message queues vs pub/sub — and Kafka, deeply
First principles: synchronous calls couple availability (B down → A fails) and latency (A waits for B). A durable log in between decouples them: the producer writes and moves on; the consumer processes at its own pace; bursts get absorbed; retries become replays.
- Queue (work distribution): each message consumed by exactly ONE worker. Job processing.
- Pub/Sub (fan-out): each message delivered to ALL subscribers. Event notification.
- Kafka does both: within a consumer group = queue semantics (partitions divided among members); across groups = pub/sub (every group gets every message).
Kafka anatomy — know this cold, it's your production system
TOPIC "events" (partitioned, each partition = ordered append-only log)
P0: [0][1][2][3][4][5][6] ← producer appends; offset = position
P1: [0][1][2][3][4]
P2: [0][1][2][3][4][5]
CONSUMER GROUP "opensearch-ingest"
consumer-A ← P0, P1 partitions are divided among
consumer-B ← P2 group members; one partition
never has 2 consumers in a group
Another group "billing-agg" ALSO reads all partitions,
with its OWN offsets — fan-out without interference.| Concept | What it is | The design consequence |
|---|---|---|
| Partitions | The unit of parallelism AND ordering; ordering is guaranteed only within a partition | Choose the partition key so things that must be ordered share a key (e.g., user_id → all of a user's events in order). Max useful consumers in a group = partition count. |
| Consumer groups | Kafka tracks (group, partition) → committed offset | Add a consumer → rebalance redistributes partitions (a brief pause — a real operational event worth mentioning). |
| Offsets & commit discipline | Consumers commit offsets after processing | Commit before processing → crash loses messages (at-most-once). Commit after → crash reprocesses (at-least-once). This tiny ordering decision IS the delivery semantics. |
| Consumer lag | Latest offset − committed offset | THE health metric for any pipeline; alert on lag growth, scale consumers (up to partition count) to drain. |
Delivery semantics
| Semantics | How | Cost | Use |
|---|---|---|---|
| At-most-once | Commit, then process | May lose messages | Metrics where loss is OK |
| At-least-once | Process, then commit | Duplicates on retry — need idempotent consumers | The industry default |
| Exactly-once | Kafka transactions / idempotent producer + txn consumer — real but narrow (Kafka→Kafka); across arbitrary external systems it's effectively at-least-once + idempotency | Complexity, throughput | Billing, ledgers |
The interview-winning sentence: "In practice we build at-least-once delivery with idempotent processing — that's cheaper and more robust than chasing true exactly-once across system boundaries."
YOUR HOOK: "This is my daily system: Kafka event pipelines feeding OpenSearch with batched ingestion, serving 5,000+ concurrent users across 20+ microservices. We consume in batches and bulk-index into OpenSearch — one bulk call per N events instead of N calls — because per-document indexing collapses at that event rate. We watch consumer lag as the pipeline health metric, and consumers are idempotent because at-least-once redelivery is a fact of life (doc IDs make OpenSearch upserts naturally idempotent)."
10Rate limiting algorithms
You BUILT this on Kong. Own this topic. Per-tenant rate limiting for 50+ enterprise clients is your credential — every algorithm below should end with "…and here's what we actually did."
Fixed window
Counter per window (100/min); reset at boundary. Trivial, O(1) memory. Boundary burst: 100 at 0:59 + 100 at 1:01 = 200 in 2s.
Sliding window log
Store the timestamp of every request; count the last 60s. Exact, but O(requests) memory — expensive.
Sliding window counter
Weighted blend: prev_window × overlap% + current. ~Accurate, O(1); it's an approximation.
Token bucket
Bucket of capacity B, refills r tokens/s; a request costs ≥1 token. Allows controlled bursts (bucket size) at sustained rate r; per-request cost is tunable. Two params to tune.
Leaky bucket
Queue drained at a fixed rate. Perfectly smooth output; adds queueing latency — bursts wait or drop.
TOKEN BUCKET capacity B=10, refill r=5/s
refill 5/s ──────────┐
v
+-------------+
| ● ● ● ● ● | ← 5 tokens now
+-------------+
request arrives ──> token available? ── yes → take 1, allow
│
no → 429 + Retry-AfterDistributed rate limiting (the real problem): counters must be shared across gateway replicas → Redis with atomic ops (INCR + EXPIRE, or a Lua script for check-and-decrement in one round trip). Tradeoff to name: exact global counting (Redis on the hot path — adds latency, Redis becomes a dependency) vs local counters synced async (fast, slightly leaky). Also name the response contract: 429, Retry-After, X-RateLimit-Limit/Remaining/Reset.
YOUR HOOK: "I implemented per-tenant rate limiting on Kong for 50+ enterprise clients. Two extra wrinkles from LLM serving: (1) limits should be token-based, not just request-based — one request can cost 100x another in compute, so we meter usage in LLM tokens; (2) limits are per-tenant with different tiers, so config is data, not code." (Full design: file 05a.)
11API gateway pattern
One front door for all services: auth (API keys/JWT/OAuth), rate limiting, routing, TLS termination, request/response transforms, usage metering/analytics, caching, WAF. Services behind it stay simple and consistent — cross-cutting concerns live once.
+---------------------------+
50+ tenants ──────> | KONG (gateway) |
| authn → rate limit → |
| route → meter (billing) |
+---------------------------+
| | |
[svc A] [svc B] ... [svc T] 20+ services on K8sTradeoffs to volunteer: the gateway is a single point of failure (run N replicas behind an L4 LB) and a potential bottleneck/latency tax (keep per-request work O(1): cache auth lookups, async-flush metering). Distinguish it from a service mesh (east–west, sidecar-based) — the gateway is north–south.
YOUR HOOK — this is literally your job: "I run Kong on Kubernetes as the front door for an LLM API platform — auth, per-tenant rate limits, and billing metering for 50+ enterprise clients at 2M+ requests/month." Deliver it as your credential sentence whenever gateways come up.
12CAP theorem — honestly — plus PACELC
Statement: when a network Partition happens, a distributed system must choose: refuse some requests (Consistency — every read sees the latest write) or serve possibly-stale data (Availability).
The honest version (say this — most candidates get it wrong):
· CAP is only about behavior during partitions. A "CA system" is basically meaningless for a distributed system — partitions WILL happen; the only choice is C-vs-A when they do.
· C here means linearizability — a much stronger claim than "ACID consistency."
· Real systems aren't globally "CP" or "AP" — they choose per operation. Your billing writes want CP behavior; your search reads are happily AP.
PACELC completes it: if Partition: A vs C; Else (normal operation): Latency vs Consistency. Even with zero partitions, you pay latency for consistency — sync replication waits for follower acks; quorum reads add round trips. This is the everyday tradeoff; partitions are the rare one.
- DynamoDB/Cassandra: PA/EL (available + fast, eventual).
- Spanner/CockroachDB, ZooKeeper/etcd: PC/EC (consistent, pays latency).
- Postgres + async replicas: primary reads are consistent; replica reads trade freshness for latency/scale — PACELC live in your stack.
YOUR HOOK: "Billing for 50+ enterprise clients is our C side — an undercounted invoice is a real cost, so usage events are durably written before we ack. Search over OpenSearch is our A/EL side — near-real-time indexing means seconds of staleness, which users never notice."
13Idempotency
Definition: doing an operation N times has the same effect as once. Why it's foundational: networks fail ambiguously — a timeout doesn't tell you whether the operation happened. The only safe response is retry, and retry is only safe if the operation is idempotent. Idempotency is what makes at-least-once delivery (Kafka), retries (file 03), and safe payment flows possible.
Idempotency keys
Client sends a unique key per logical operation; server stores key → result; duplicate key → return the stored result, don't re-execute. Stripe's API popularized this — you integrate Stripe/Razorpay; their retry-safety contract is exactly this.
Natural idempotency
SET x=5 (vs INCR x), upserts by deterministic ID (your OpenSearch doc-ID upserts), state machines that ignore already-applied transitions (if status != 'pending': return).
Dedup at the sink
Unique constraint on event_id in Postgres — the DB enforces exactly-once effect atop at-least-once delivery.
YOUR HOOK: "Payments made this concrete for me: with Stripe/Razorpay, a timeout on a charge call is ambiguous — you MUST retry with an idempotency key or risk double-charging. And our Kafka consumers are idempotent by construction since redelivery is guaranteed to happen eventually."
14Webhooks vs polling
The question: how does a client learn that something happened in another system?
| Polling | Webhooks (push) | |
|---|---|---|
| How | Client asks every N sec | Server POSTs to client's URL on event |
| Latency | Up to N sec | ~Immediate |
| Load | Wasteful — most polls return nothing | Proportional to events |
| Client needs | Nothing special | A public HTTPS endpoint |
| Reliability burden | On client (just poll again) | On SERVER: retries w/ backoff, ordering, dedup; client must verify signatures + be idempotent |
Middle grounds: long polling (server holds the request until data/timeout), SSE (server→client stream over HTTP — what LLM APIs use for token streaming), WebSockets (bidirectional, for chat/collab).
Webhook provider checklist (a mini design question in itself — file 05d): sign payloads (HMAC) so receivers can verify; retry with exponential backoff on non-2xx; dead-letter after N attempts; deliver at-least-once → receivers dedupe on event ID; don't guarantee ordering (put sequence numbers in events instead).
YOUR HOOK: "Stripe/Razorpay webhooks are my lived version: verify the HMAC signature, return 2xx fast (enqueue, don't process inline — their timeout is short), and handle duplicate delivery idempotently because their retries WILL redeliver."
15Cheat sheet — block → your production story
One line per block, ready to deliver. Memorize the right column — it's your credibility in table form.
| Block | Your one-liner |
|---|---|
| LB / L7 | Kong + K8s Services front the whole platform |
| Stateless scaling | 20+ microservices, HPA on K8s, state in PG/Redis/Kafka |
| Replication | PG primary+replicas; fresh reads→primary, analytics→replicas |
| Kafka | Event pipeline → batched OpenSearch bulk ingestion, 5K+ concurrent users |
| Delivery semantics | At-least-once + idempotent consumers (doc-ID upserts) |
| Rate limiting | Built per-tenant limits on Kong for 50+ enterprise clients |
| API gateway | Kong: auth, rate limit, billing metering @ 2M+ req/month |
| Caching | Redis: rate-limit counters, auth/session, hot config |
| CAP/PACELC | Billing=consistent+durable; search=eventual+fast |
| Idempotency | Stripe/Razorpay idempotency keys; dedup on event IDs |
| Webhooks | Consume Stripe/Razorpay webhooks: verify, enqueue, dedupe |
Q&A — probe yourself the way they will
Your 4-shard system uses hash(key) mod 4 and you need a 5th shard. What happens, and what's the fix?
Going mod 4 → mod 5 remaps ~80% of keys — a massive live-traffic data migration. The fix is consistent hashing: nodes and keys live on a ring, a key belongs to the first node clockwise, so adding a node moves only ~1/N of keys (the arc it takes over). Add virtual nodes so the load spreads evenly. Bonus point: Redis Cluster's 16384 hash slots are the same idea, discretized.
A consumer group has 8 consumers on a 6-partition topic. What happens?
Two consumers sit idle — one partition never has two consumers in a group, so max useful consumers = partition count. If you need more parallelism, add partitions (carefully — it changes key→partition mapping for future messages) or do work fan-out downstream. Also mention: adding/removing consumers triggers a rebalance, a brief pause that is a real operational event.
A hot cache key expires and the database melts. Name the failure mode and two fixes.
Cache stampede / thundering herd: the moment the key expires, thousands of concurrent misses all rebuild from the DB. Fixes: per-key mutex / single-flight (one request rebuilds, the rest wait or serve stale), stale-while-revalidate (serve old value while refreshing in the background), and jittered TTLs so hot keys don't expire in sync.
Can you get exactly-once delivery from Kafka through your API to Stripe?
Not literally — exactly-once is real but narrow (Kafka→Kafka transactions). Across arbitrary external systems the honest engineering answer is at-least-once delivery + idempotent processing: consumers commit after processing (so redelivery is possible), and every side effect is idempotent — Stripe charge calls carry an idempotency key, OpenSearch writes are doc-ID upserts, Postgres dedupes on a unique event_id. Same observable result as exactly-once, far cheaper.
During Postgres failover, the old leader comes back online. What's the risk and how is it prevented?
Split brain: the old leader still thinks it's leader, so two nodes accept writes and the data diverges. Prevention: fencing (STONITH — forcibly shut the old node out, e.g. revoke its storage/network access) and quorum-based leader election so only one node can hold leadership at a time. Also name the second failover danger: async-replicated writes that never reached the promoted follower are simply lost.
Token bucket vs leaky bucket for an LLM API gateway — which do you pick and why?
Token bucket. LLM traffic is naturally bursty and per-request cost varies ~100x, and token bucket handles both: bucket capacity B permits controlled bursts, and you can charge each request a variable number of tokens — meter in LLM tokens, not requests. Leaky bucket forces perfectly smooth output, which adds queueing latency that streaming clients feel immediately. Then say the distributed part: counters live in Redis with atomic Lua check-and-decrement, and the response contract is 429 + Retry-After + X-RateLimit-* headers.
Self-test — close the tab and answer these
Blocks 1–5: traffic and data
- L4 vs L7 — what can each see, and which one is Kong?
- Why is least-connections the right LB algorithm for LLM streaming?
- Recite the scaling order: stateless tier vs stateful tier.
- Sync vs async replication: who acks when, and what does each lose?
- What is replication lag and name two read-your-own-writes fixes.
- Range vs hash vs directory sharding — one pro and one con each.
- Why is
tenant_ida good shard key, and what's the whale-tenant fix?
Blocks 6–10: hashing, caching, queues, limits
- Draw the consistent hashing ring from memory; explain vnodes in one sentence.
- Exactly what fraction of keys moves when a node joins a ring of N nodes?
- Cache-aside vs write-through vs write-behind — one line each.
- Name the three classic cache failure modes and one fix for each.
- Why is ordering only guaranteed within a Kafka partition, and how do you exploit that with a partition key?
- Commit-before vs commit-after processing: which delivery semantics does each produce?
- What is consumer lag and why is it THE pipeline health metric?
- All five rate-limiter algorithms with their signature flaw or superpower.
Blocks 11–15: gateway, CAP, idempotency, webhooks
- List six cross-cutting concerns an API gateway owns.
- Gateway vs service mesh — north-south vs east-west in one sentence.
- Say the honest CAP version: what is CAP actually about, and why is "CA" meaningless?
- Expand PACELC and give one PA/EL system and one PC/EC system.
- Three ways to build idempotency (keys, natural, dedup-at-sink) with your production example for each.
- The webhook provider checklist — five items.
- Deliver your Kong credential sentence and your Kafka pipeline one-liner cold.