Your systems as interview answers

"This is close to what I run in production, so let me describe how I'd design it — which is also how our system actually works.

The platform serves LLM APIs to 50+ enterprise clients at 2M+ requests a month. Three concerns have to be handled on every single request — who are you, are you allowed to make this call right now, and what should it cost you — and the worst design is scattering those across services. So the first architectural decision is a gateway layer: we run Kong on Kubernetes as the single front door, and behind it the platform is 20+ microservices that stay clean because auth, rate limiting, and metering live in one place.

Walking a request through: it hits Kong, which resolves the API key to a tenant — that lookup is cached in Redis, because adding a database read to every request would tax the whole platform. Then per-tenant rate limiting, which I built: limits are per-tenant and tiered, so the config is data, not code — an enterprise client's limits differ from a smaller one's, and changing them shouldn't need a deploy. The counters live in Redis so all gateway replicas share state — rate limiting on a single node's memory breaks the moment you scale horizontally.

Then the request routes to the right backend service. The services are stateless — all state is in PostgreSQL, Redis, or the event pipeline — which is what lets Kubernetes scale and reschedule them freely.

Now billing, which is the interesting part. The rule I design by: the billing record must be durable, but billing must never sit in the request path. If metering were synchronous, a billing-system slowdown would slow every API call — coupling revenue plumbing to user latency. So each request emits a usage event asynchronously; events flow through our Kafka pipeline and are aggregated into PostgreSQL as the usage ledger. Because the pipeline delivers at-least-once, the aggregation is idempotent — dedupe on event ID — so retries and replays can't double-bill anyone. That ledger drives invoicing, and we charge through Stripe and Razorpay — and working with payment APIs is where I really internalized idempotency: a timed-out charge call is ambiguous, so every mutation carries an idempotency key, and their webhooks are verified, acked fast, and processed idempotently because their retries will redeliver.

Failure thinking: the gateway is the obvious critical point, so it's N replicas — the platform has no single-box front door. If Redis degrades, rate limiting fails open — for a paid API, blocking all 50 tenants because the limiter's dependency blinked is the worse outage; we allow, log loudly, and alert. And billing events are the one thing we refuse to lose — durability there is the design's hard constraint."

"I run exactly this shape in production, so let me design it the way it actually works. The setup: 5,000+ concurrent users generating events across 20+ microservices, and those events need to end up searchable in OpenSearch.

The naive version is each service writing directly to OpenSearch as events happen. That fails in three distinct ways. Coupling — twenty services now each carry an OpenSearch dependency; if it's slow or down, twenty services feel it. Write pattern — thousands of individual index operations per second is the worst way to feed a search cluster; per-document indexing has fixed overhead, and the cluster spends itself on overhead instead of work. Burst behavior — user activity is spiky, and a synchronous design transfers every spike straight onto the search cluster.

So the design is: Kafka in the middle, batching at the sink.

Producers — the 20+ services — do one cheap, durable thing: append the event to Kafka and move on. Their latency and availability are now decoupled from the search cluster entirely; OpenSearch could be down for an hour and no user-facing service notices — events wait in the log.

On the other side, consumers read events in batches and use OpenSearch's bulk API — one bulk request carrying a large batch of documents instead of that many individual writes. The batching policy is size-or-time — flush when the batch is full or when a short timer fires — so under high traffic you get maximal efficiency and under low traffic events still land promptly rather than waiting for a batch to fill. That knob is a latency-versus-throughput dial, and it's tunable after deployment, which matters.

Correctness: Kafka gives at-least-once delivery — a consumer that crashes after indexing but before committing offsets will reprocess. So the writes are idempotent: deterministic document IDs make re-indexing an upsert — replay-safe by construction. And ordering: Kafka guarantees order only within a partition, so we partition by the key that users actually perceive order in — events for the same entity share a partition; global ordering is a cost nobody should pay for.

Operationally, the health metric for the whole pipeline is consumer lag — the gap between what's produced and what's consumed. Lag growing means consumers are falling behind: scale consumers up to the partition count, or tune batch sizes. And bursts get absorbed as visible, temporary lag rather than as a search-cluster meltdown — the log is the backpressure mechanism. A poison message — malformed event that always fails — must not stall the partition behind it, so after bounded retries it goes to a dead-letter topic with an alert, and we replay after fixing the bug; idempotent writes make replays safe."

"We had an endpoint in production taking 71 seconds. In a system of 20+ microservices, that's the worst kind of problem — the slowness could be anywhere in a chain of hops, and everyone's instinct suggests a different culprit. The lesson I took from this one is: don't debug by intuition; make the system show you where the time goes. So I went to distributed tracing.

Step one: get the shape of the problem. A trace of one slow request gives you the span waterfall — every hop and operation, each with its duration, as a tree. The first question is never 'what's wrong,' it's 'which span owns the time?' Seventy-one seconds spread thinly across many spans is a systemic problem; seventy-one seconds concentrated in one span is a local one. Reading the waterfall collapsed the search space from 'twenty services' to one place — that's the whole value of tracing: it turns a distributed mystery into a local question.

Step two: read the shape inside the culprit. Zooming into that span's children, the structure told the story — the time wasn't one expensive operation, it was an access pattern: many small sequential operations where the work should have been batched. Serial round trips stack linearly — do enough of them in a row and per-call overhead alone produces a colossal total. Nothing was 'broken'; the pattern was wrong. In my experience that's the most common production performance bug: not slow components, but chatty patterns between fast components.

Step three: fix the pattern, not the symptom. The fix was to restructure the access pattern — batch the work, eliminate redundant round trips — turning many serial calls into few. I deliberately did not start by adding caching or scaling hardware, because both would have masked the underlying pattern and left the real cost in place.

Step four: verify with the same instrument. Same endpoint, same tracing: 2 seconds — about a 35x improvement — and the new waterfall looked healthy, a short, flat profile instead of a long staircase. Measuring with the same tool that found the problem closes the loop; 'it feels faster' isn't verification.

What it changed about how I work: I now treat traces as the first move on any latency issue, not the last resort — and it's why, when I design systems, trace propagation goes in on day one, at the gateway. When this endpoint misbehaved, the instrumentation determined whether diagnosis took thirty minutes or a week."