03 · System Design — 04
Six Classic Designs, SDE-2 Depth
For each design: requirements → estimation → API → high-level design → deep dive → what interviewers actually probe. Practice each aloud in ~25 minutes. Where a design touches something you run in production (rate limiter, queues, batching, gateway), the Your edge note tells you how to cash that in.
(a)URL Shortener
The "hello world" of system design — interviewers use it to check whether you can run the full framework cleanly, and whether you understand read-heavy systems + caching + key generation.
Requirements
Functional: shorten long URL → short code; redirect short → long; optional custom alias; optional expiry. Out of scope (say it): analytics dashboard, auth-heavy features.
Non-functional: redirects must be FAST (<50ms server-side) and highly available — the shortener is in the critical path of someone else's click. Read-heavy: assume 100:1 read:write. Codes must not be guessable-sequential (mild security), must never collide.
Estimation
100M new URLs/month → ~40 writes/s 100:1 → ~4,000 reads/s avg, ~12K peak Storage: 100M × 500 B ≈ 50 GB/month → ~3 TB over 5 yrs (fine for one DB + replicas) Conclusion: writes trivial, reads modest but latency-sensitive → CACHE-CENTRIC design.
API
POST /api/v1/urls { long_url, custom_alias?, expires_at? } → 201 { short_url }
GET /{code} → 301/302 redirect
DELETE /api/v1/urls/{code} (auth'd owner)
301 vs 302 — a classic probe: 301 (permanent) lets browsers/CDNs cache the redirect → less load on you, but you lose click visibility and can't change the mapping. 302 keeps every click flowing through you (analytics, mutable mappings). Pick 302 if analytics matter, 301 if pure scale. Saying this unprompted is an instant credibility win.
High-Level Design
Deep Dive: Short-Code Generation
The interesting problem. Options:
- Hash the URL (MD5 → take 7 chars): same URL → same code (dedup for free), but truncation collides → need retry-with-salt loop. Meh.
- Auto-increment ID → base62 encode: 627 ≈ 3.5 trillion codes. Simple, collision-free. Problems: sequential codes are enumerable (competitor scrapes your namespace) and a single counter is a bottleneck/SPOF.
- Counter ranges (the clean answer): a coordination service hands each API server a block of IDs (server A: 1M-2M, server B: 2M-3M). Servers allocate from their block in memory — no per-write coordination; base62-encode; optionally bit-scramble so codes don't look sequential. Lost block on crash = wasted range, who cares (3.5T namespace).
- Pre-generated key pool: offline job fills a
keystable with random unused codes; writers pop one. Simple, random codes; the pool service needs care under concurrency.
Recommend #3, mention #4 as equally valid. Collisions: with generated-unique IDs there are none by construction; with random/hashed codes, INSERT ... ON CONFLICT retry loop.
What Interviewers Probe
Q: 301 or 302 for the redirect — and what does a cached redirect do to your analytics?
301 (permanent) lets browsers/CDNs cache the redirect — less load on you, but clicks stop reaching your servers, so analytics die and you can't change the mapping. 302 keeps every click flowing through you. Pick 302 if analytics matter, 301 for pure scale. Related: cache TTL vs deleted/expired mappings — there's a serve-stale window; decide whether it's acceptable.
Q: One viral link is getting 100K reads/s. What breaks, and what do you do?
A hot key. Redis itself handles 100K reads/s of a single key fine; if it becomes a bottleneck, add a local in-process cache on the API pods (the mapping is immutable-ish, so a short TTL local cache is nearly free and removes the network hop entirely).
Q: How do you handle expired URLs at 3 TB scale?
Lazy delete on read (check expires_at at redirect time, 410/404 if past) plus a periodic sweep job — not a cron scanning 3 TB. Cache entries carry their own TTL so they age out.
Q: How would you add click analytics without slowing redirects?
Answer from your life: fire an event to Kafka asynchronously after responding; aggregate downstream. Never put analytics writes in the redirect path. Your edge: "this is exactly my event pipeline pattern — emit to Kafka, batch-ingest into an analytics store."
(b)Rate Limiter
This is your home game — you built per-tenant rate limiting on Kong. Open with that: "I've actually built this in production for 50+ enterprise clients, so let me walk through it the way we approached it."
Requirements
Functional: limit requests per client (per API key / tenant / IP) per time window; different tiers (free: 10/min, enterprise: 1000/min); return 429 + Retry-After; rules configurable without deploys.
Non-functional: ultra-low added latency (<1-2ms — it's on EVERY request), high availability, works across N distributed gateway nodes, and a decided failure policy (fail open vs closed).
Estimation
Say the platform does 10K QPS across all tenants. Each request = 1 limiter check = 1-2 Redis ops → 10-20K Redis ops/s: one Redis node handles this (100K ops/s headroom). Memory: 100K active keys × ~100B ≈ 10 MB. Trivial. ⇒ The problem is LATENCY & CORRECTNESS, not scale.
API (it's middleware, so define the contract)
Internal: allow(key, cost=1) → { allowed: bool, remaining, reset_at }
On deny: 429 + Retry-After: <sec> + X-RateLimit-Limit/Remaining/Reset
Config: rules keyed by tenant/tier/route, stored as data (DB/config svc),
hot-reloaded — NOT hardcoded.
High-Level Design
+--------------------------------------+
client --> | Gateway node 1..N (Kong) |
| [auth] → [rate-limit plugin] → route|
+-----------------│--------------------+
│ atomic check-and-decrement
v (Lua script, 1 RTT)
[ Redis (counters) ]
^
[ Config store: tenant → tier → limits ] (cached on node,
refreshed async)
Deep Dive: Algorithm + Distributed Correctness
- Algorithm choice: token bucket — allows short bursts (bucket capacity) while enforcing sustained rate (refill); O(1) memory per key (2 numbers: tokens, last_refill). Sliding-window counter is the runner-up. Fixed window only if you accept boundary bursts.
- Race condition: GET then SET from two gateway nodes both see "1 token left" and both allow → over-admission. Fix: atomic Lua script in Redis — read tokens, compute refill since last timestamp, decide, write — one atomic round trip.
-- sketch: tokens & last_ts in a hash; refill = (now-last_ts)*rate
-- if tokens >= cost then tokens -= cost; return allowed- Latency tradeoff (the SDE-2 differentiator): exact global counting puts Redis in the hot path of every request. Alternative: local counters with async sync — each node enforces limit/N locally, reconciles periodically; near-zero latency but slightly leaky under skewed routing. State the spectrum, pick per requirement: "billing-adjacent limits → exact; abuse protection → approximate is fine."
- Failure policy: Redis down → fail open (allow + log + alert) for a paid API — blocking all 50 tenants over a limiter dependency is the worse outage. Fail closed only for security-critical limits (login attempts).
What Interviewers Probe
Q: A client sends 10 requests at 11:59:59 and 10 more at 12:00:01 under a "10/min" limit. What happened, and what fixes it?
The fixed-window boundary flaw: 20 requests slipped through in 2 seconds because each burst landed in a different window. Fix with a sliding-window counter (weighted blend of adjacent windows) or a token bucket (bucket capacity caps the burst, refill enforces the sustained rate).
Q: Two gateway nodes check the same key concurrently and both admit the "last" request. How do you make the check correct across N nodes?
The atomic Lua answer: a single Redis Lua script reads the token count + last-refill timestamp, computes the refill, decides, and writes — one atomic round trip. No GET-then-SET race. For a hot tenant key that overloads one shard: shard counters per tenant, or drop to local counters with async sync.
Q: Free tier gets 10/min, enterprise 1000/min, and product wants to change these weekly. Where do limits live?
Config as data: rules keyed by tenant/tier/route in a DB or config service, cached on each gateway node and refreshed asynchronously — never hardcoded, never requiring a deploy to change.
Q: Anything request-count limits miss for modern APIs?
Your closer: "One thing production taught me: for LLM APIs, request-count limits aren't enough — one request can cost 100x another, so we also meter token-based cost and rate-limit on that." (Segue to the LLM-system designs page — often turns the rest of the interview into your strongest material.)
(c)Chat / Messaging System
Tests: real-time delivery (WebSockets), fan-out, ordering, offline delivery, storage modeling.
Requirements
Functional: 1:1 chat; group chat (cap ~200 members); delivery/read receipts; online presence; message history; offline users get messages on reconnect. Out of scope: E2E encryption details, media (mention presigned-URL upload — see design (e)), calls.
Non-functional: near-real-time (<200ms delivery same-region); no message loss (durability!); order preserved per conversation; scale: 10M DAU.
Estimation
10M DAU × 40 msgs sent/day = 4×10^8 msgs/day ≈ 5K msgs/s avg, 15K peak. Msg ~200 B + metadata → ~80 GB/day raw → ~30 TB/yr ⇒ partitioned store, not one PG. Concurrent connections: 10-20% of DAU online ≈ 1-2M open WebSockets → at ~100K-200K conns/node ⇒ 10-20 chat servers + a routing layer.
API / Protocol
WebSocket after auth: wss://chat.example.com/connect (JWT)
C→S: { type:"send", client_msg_id, conv_id, body }
S→C: { type:"ack", client_msg_id, msg_id, seq }
S→C: { type:"message", conv_id, msg_id, seq, sender, body, ts }
REST for the rest:
GET /conversations/{id}/messages?before_seq=...&limit=50 (history, cursor)
POST /conversations | GET /users/{id}/conversations
Why WebSocket not polling: 1-2M clients polling every 2s = 500K-1M QPS of mostly-empty responses. Push inverts that: traffic proportional to actual messages.
High-Level Design
mobile/web ──WS──> [ LB (L4, sticky) ] ──> [ Chat servers 1..N ]
│ (hold WS conns; STATEFUL-ish)
┌─────────────────────────┤
v v
[ Session registry (Redis): [ Kafka: topic "messages",
user_id → chat_server_id ] partitioned by conv_id ]
│
┌───────────────┼───────────────┐
v v v
[ Delivery workers ] [ Persist workers ] [ Push notif svc ]
│ lookup recipient's │ → message store (offline users)
│ server, forward v
└──WS push──> user [ Cassandra/partitioned store:
PK (conv_id, seq) ]
Flow: sender → their chat server → produce to Kafka (durable BEFORE ack — no loss) → persist worker writes store → delivery worker looks up each recipient's server in the session registry → pushes over their WebSocket → offline? → push notification + they page history on reconnect.
Deep Dive: Ordering + Delivery Guarantees
- Ordering: global ordering is unnecessary and expensive; users only perceive per-conversation order. Partition Kafka by
conv_id(per-partition order = per-conversation order) and assign a per-conversation sequence number at write. Clients render byseq, detect gaps → fetch missing via history API. - No-loss + dedup: server acks the sender only after the Kafka write (durable). Sender retries unacked sends with the same
client_msg_id; server dedupes → at-least-once transport + idempotency = effectively exactly-once UX. (Your everyday Kafka discipline, applied.) - Receipts: delivery/read receipts are just tiny messages flowing the same pipeline (batched — read receipts in a 200-group would otherwise 200x traffic).
- Presence: heartbeat every ~30s → TTL'd Redis key; expiry = offline. Don't broadcast every flap; subscribers pull on view.
What Interviewers Probe
Q: A chat server holding 150K WebSocket connections dies. What do users experience, and what do you lose?
Clients auto-reconnect to any node, re-register in the session registry, and page missed messages by last-seen seq. Statefulness is recoverable — connections are lost, messages never are, because a message never lives only in a chat server's memory (it's in Kafka before the sender is even acked).
Q: How does a message to a 200-member group get stored and delivered?
1 Kafka write, 200 deliveries — fan-out happens at delivery, not storage. Store once per conversation, not per recipient. (Contrast with feeds, where the hybrid design fans out at write time for normal users — different read patterns justify different fan-out points.)
Q: Why Cassandra (or similar) for messages instead of Postgres?
Write-heavy, append-mostly workload with a natural partition key (conv_id, clustering by seq), and 30 TB/yr outgrows single-node PG. Postgres sharded by conv_id is a defensible alternative — say you'd start there at smaller scale.
Q: How do you guarantee messages aren't duplicated when the sender retries?
Sender retries unacked sends with the same client_msg_id; server dedupes on it. At-least-once transport + idempotency key = effectively exactly-once UX. Your edge: the middle of this design — Kafka partitioned by key, ordered consumption, idempotent processing, consumer lag as the health metric — is literally your production pipeline. Say so.
(d)News Feed
Tests: fan-out-on-write vs fan-out-on-read — the single most famous tradeoff in system design interviews. Also ranking, celebrity problem, cache design.
Requirements
Functional: users post; users follow; feed = recent posts from followees, reverse-chron (mention ranking as pluggable later); like/comment counts displayed. Out of scope: ads, stories, the ranking ML itself.
Non-functional: feed load p99 < 200ms (THE user experience); read-heavy ~100:1; eventual consistency fine (a post appearing after 10-30s is acceptable — SAY this, it unlocks the whole design); scale: 10M DAU.
Estimation
Posts: 10M DAU × 1 post/day ≈ 120 writes/s (peak ~500) — small! Feed loads: 10M × 10 loads/day ≈ 1,200 QPS avg, ~5K peak — the real load. Follow graph: avg 200 followees. Celebrity: 10M followers (outlier that breaks naive designs). ⇒ optimize READS. Precompute feeds.
API
POST /posts { text, media_ids? } → 201
GET /feed?cursor=...&limit=20 → { posts[], next_cursor }
POST /users/{id}/follow | DELETE .../follow
High-Level Design (hybrid fan-out — the correct answer)
WRITE PATH (post creation)
author ─> [API] ─> [Posts DB] ─> Kafka "new_post" ─> [Fan-out workers]
│ for each follower:
│ LPUSH feed:{follower_id} post_id
│ LTRIM to ~800 entries
v
[ Redis feed cache per user ]
CELEBRITY (>~100K followers): SKIP fan-out; mark author in "hot authors" set
READ PATH (feed load)
user ─> [API] ─> merge: precomputed list ∪ recent posts from followed
Redis feed:{user_id} celebrities (queried at read
│ time, small set, heavily cached)
v
hydrate post_ids → [posts cache → Posts DB] → rank → return
Deep Dive: The Fan-out Tradeoff (rehearse this monologue)
"Fan-out-on-read (pull): store posts once; on feed load, query 200 followees' recent posts and merge. Writes are cheap; reads are expensive — 200 lookups per feed load × 5K QPS = 1M lookups/s. Latency budget dead.
"Fan-out-on-write (push): when someone posts, insert the post ID into every follower's precomputed feed list. Reads become one Redis list read — fast. But a 10M-follower celebrity post = 10M writes for one action — write amplification that can take minutes and hammer the cluster; and inactive users get feeds computed for nothing.
"So: hybrid. Push for normal users (bounded fan-out, ~200 avg followers), pull for celebrities (bounded pull set — you follow maybe 5 celebrities), merge at read. Each strategy where its cost is bounded."
Supporting details: feed lists in Redis capped (~800 ids, LTRIM) — full history stays in the DB, feed cache is a view; fan-out via Kafka workers = async, absorbs the burst, retryable (your pipeline pattern again); new follow → backfill on next read (lazy), unfollow → filter at read + lazy repair.
What Interviewers Probe
Q: Where exactly is the push/pull threshold — what follower count makes someone a "celebrity"?
It's empirical, roughly 10K-100K followers. The honest answer: measure fan-out latency and cap it — the threshold is "the follower count at which fan-out no longer completes within your freshness budget," not a magic number.
Q: A user posts and their friend doesn't see it for 20 seconds. Bug?
No — you set this expectation in requirements: eventual consistency is fine for feeds; a post appearing after 10-30s is acceptable. This is why fan-out can be async via Kafka workers at all. Saying it upfront unlocks the whole design.
Q: How do you render like counts at 5K feed loads/sec?
Don't COUNT(*) per render — counter cache in Redis, batched/periodic flush to the DB; accept approximate counts. (Batching to protect a store = your OpenSearch pattern.)
Q: How would you add ranking without rebuilding the system?
Fetch 3-5x candidates, score them (recency/affinity/engagement or an ML model behind a service), return the top N — keep it pluggable at the read path's merge step, and don't get dragged into the ML itself.
Q: Why cursor pagination for the feed instead of offset?
Offset breaks when new items land on top — page 2 shows duplicates or skips items. Cursor by post_id/timestamp is stable under insertion. Never offset for a mutating list.
(e)File Upload Service (Dropbox/Drive-lite)
Tests: whether you know to keep large blobs OUT of your API servers and DB — presigned URLs, chunking, metadata/data separation.
Requirements
Functional: upload files up to 5 GB; download; list/organize; share via link; resumable uploads on flaky networks. Out of scope unless asked: real-time sync/conflict resolution (that's the harder Dropbox interview), folder permissions depth.
Non-functional: durability is the headline — never lose a file (object storage: 11 nines); upload throughput limited by user bandwidth, not us; resumability for large files; scale: 1M users.
Estimation
1M users × 2 uploads/day × avg 5 MB = 10 TB/day ingest ⇒ object storage (S3/GCS), never through your DB — and ideally not through your API servers either: 10 TB/day of proxied bytes = pointless compute + egress. ⇒ PRESIGNED URLS. Metadata: 2M files/day × ~500 B = 1 GB/day — Postgres is fine for years.
API
POST /files/initiate { name, size, mime, chunk_size }
→ { upload_id, chunk_presigned_urls[] } (or one URL if small)
PUT <presigned S3 URL per chunk> (client → S3 DIRECT)
POST /files/{upload_id}/complete { parts: [{n, etag}] }
GET /files/{id}/download → 302 to presigned GET URL (time-limited)
GET /files?folder=...&cursor=...
POST /files/{id}/share → { link_token, expires_at }
High-Level Design
control plane (small requests) data plane (bytes)
client ──> [API svc] ──> [Postgres: files, chunks, client ══════════> [ S3 ]
^ │ uploads, shares] PUT chunk 1..N │
│ │ issue presigned URLs (parallel, │
│ └──────────────────────────────────────> resumable) │
│ │
└── complete ──> [API svc] verify parts/etags ──> mark COMPLETE │
│ │
└─> Kafka "file_uploaded" ─> [workers: virus scan,│
thumbnails, search indexing] <───────────────┘
The core idea to articulate: separate the control plane from the data plane. Your servers handle small metadata requests and mint capability-scoped, time-limited presigned URLs; the heavy bytes flow client↔S3 directly. Your API tier scales with request count, not gigabytes.
Deep Dive: Chunking + Resumability
- Split 5 GB into 5-10 MB chunks (S3 multipart: 5 MB min part). Benefits: resume (connection drops at 4.9 GB → re-upload one chunk, not everything — client asks "which parts are done?" and continues), parallelism (4-8 chunks concurrently saturates the pipe), integrity (per-chunk checksum/etag; verify at complete).
- State machine in metadata DB:
INITIATED → UPLOADING(parts bitmap) → COMPLETE / ABORTED. Janitor job aborts stale multipart uploads (they cost storage!). - Dedup (bonus point): hash chunks (content-addressed); identical chunk already stored → skip upload. Mention block-level dedup as the Dropbox trick; note the tradeoff (hash lookup RTT per chunk, privacy considerations).
- Post-processing is async via events — never make the user's "complete" call wait on virus scanning. (Your Kafka-worker pattern, again.)
What Interviewers Probe
Q: Why presigned URLs? Why not just upload through your API servers?
The 10 TB/day answer: proxying that through your API tier is pointless compute and egress cost, and it couples your server scaling to gigabytes instead of request count. Mint capability-scoped, time-limited presigned URLs; bytes go client↔S3 directly.
Q: An upload dies halfway and the client calls complete twice. What happens?
Mid-upload death: the parts bitmap knows which chunks landed; the client resumes only the missing ones, and a janitor job aborts stale multipart uploads so abandoned parts don't cost storage forever. Duplicate complete: idempotent by upload_id — the second call returns the same result, no double processing.
Q: Downloads are hammering S3 for a popular shared file. Now what?
CDN in front of S3 for hot/public files; signed CDN URLs for private ones. Your download endpoint is already a 302 to a presigned URL, so swapping the target for a CDN URL is transparent to clients.
Q: How do share links stay secure?
Unguessable token (high-entropy random), expiry, revocation (delete the token row), optional password. The link is a capability — treat it like one.
Q: The interviewer pushes toward real-time sync between devices. Do you take the bait?
Sketch it, don't drown in it: per-device cursor + change log, flag conflict resolution as the genuinely hard part (that's the harder Dropbox interview), and timebox it explicitly.
(f)Notification System
Tests: multi-channel fan-out, queues, retries/DLQ, idempotency, rate control, preferences. This design is basically an anthology of the building-blocks and reliability pages — and very close to your production patterns.
Requirements
Functional: send notifications over push (APNs/FCM), email, SMS, in-app; triggered by other services via API/events; user preferences & opt-outs (per channel, per category); templates; scheduled sends; batch/campaign sends. Out of scope: building an SMTP server — we use providers (SES, Twilio, FCM).
Non-functional: OTP-class notifications = seconds + at-least-once + never dropped; marketing = minutes fine + strictly respect opt-outs + rate-shaped; no duplicate sends (idempotency); scale: 50M notifications/day; graceful handling of provider outages/limits.
Estimation
50M/day ≈ 600/s avg — but campaigns are SPIKY: "send to 5M users now" = a burst that must be queued and drained at provider-safe rates, not blasted. ⇒ queue-centric, rate-shaped architecture, per-channel workers. Provider caps (e.g. SMS throughput) are the real constraint, not our compute.
API
POST /v1/notifications
{ idempotency_key, user_id (or segment_id), category, channel_hints?,
template_id, data{}, priority: "transactional"|"marketing", send_at? }
→ 202 { notification_id } (ALWAYS async — 202, not 200)
GET /v1/notifications/{id} → status per channel
PUT /v1/users/{id}/preferences { email:{marketing:false}, ... }
High-Level Design
services ──> [ Notification API ]───────────────┐
(or events from Kafka directly) │ validate, dedupe
v (idempotency_key)
[ Kafka: "notif_requests" ]
│
[ Orchestrator workers ]
│ load prefs+opt-outs, pick channels,
│ render template, split per channel
┌───────────────┼────────────────┬─────────────┐
v v v v
[Kafka: push_q] [Kafka: email_q] [Kafka: sms_q] [in-app store]
│ │ │
[Push workers] [Email workers] [SMS workers] ← per-channel
│ rate-shaped to provider limits (token bucket!)
v v v
[FCM/APNs] [SES/...] [Twilio/...]
│ failures: retry w/ backoff → after N → [DLQ + alert]
└── delivery receipts/webhooks ──> [Status store (PG)]
Deep Dive: Exactly-Once-ish Delivery + Priority
- Dedup at two layers: API layer dedupes on client
idempotency_key(caller retries safely); worker layer checks asent(notification_id, channel)record before calling the provider — because Kafka redelivery WILL happen (at-least-once), and the provider call is the side effect you must not repeat. Sending is not idempotent at the provider, so you build the idempotency around it. - Priority isolation (bulkheads!): transactional (OTP) and marketing traffic on separate topics + separate worker pools — a 5M-user campaign must never queue a login OTP behind it. This is the bulkhead pattern with an obvious business justification.
- Rate shaping: per-provider token bucket in workers (SMS provider allows X/s → drain at X/s regardless of queue depth). Backpressure lives naturally in Kafka: the campaign sits durably in the log; lag is visible; nothing melts. (Your architecture, verbatim.)
- Provider failure: circuit breaker per provider; fallback provider for email/SMS where possible; else queue accrues + alert. Delivery receipts arrive via provider webhooks → verify signature, update status, dedupe.
What Interviewers Probe
Q: A user got the same OTP twice. Walk me through every place a duplicate could come from and how you stop it.
The two-layer dedup answer: (1) caller retried the API → deduped on client idempotency_key at the API layer; (2) Kafka redelivered to a worker → worker checks the sent(notification_id, channel) record before the provider call. The provider call is the non-idempotent side effect, so you build idempotency around it yourself.
Q: Marketing fires a 5M-user campaign at 9am. What keeps login OTPs from arriving at 9:40?
Bulkheads: transactional and marketing traffic ride separate topics with separate worker pools, so the campaign can never queue an OTP behind it. The campaign itself sits durably in Kafka and drains at provider-safe rates via per-provider token buckets — queue depth is visible as lag, nothing melts.
Q: Where do you check opt-outs, and why does it matter more than most checks?
At orchestration time — a fresh read (cached briefly), not a stale copy from enqueue time. It's legally load-bearing (spam/consent law), so also log the decision for an audit trail.
Q: An email send fails. Retry?
Classify the failure: hard bounce = permanent, don't retry (and suppress the address); timeout/5xx = transient, retry with backoff, then DLQ + alert after N attempts. Blind retries on permanent failures burn provider reputation.
Q: A user is reachable on push, email, and in-app. Which channel wins?
Preference + category policy decides: in-app always; escalate to push if unread after a window (mention the escalation ladder, don't build it). Your edge: "This is structurally my daily system — Kafka in the middle, idempotent workers, DLQs, batched writes, provider webhooks with signature verification (I do this with Stripe/Razorpay), and per-tenant rate shaping like I run on Kong."
§Cross-Design Patterns (notice how few ideas there actually are)
Decouples producers from slow/bursty consumers → chat, feed fan-out, upload post-processing, notifications. (You run this.)
When read:write is high → feed lists, URL cache.
≈ exactly-once effect → chat send, notifications, payments. (You run this.)
Separate them → presigned uploads, gateway + services.
Feed list length, retry counts, chunk sizes, rate limits. Unbounded = outage pending.
The key users perceive order/locality in — conv_id, user_id, tenant_id.