Messaging | System Design Interview Handbook

🧵

Why Async At All

a buffer between producer and consumer

A synchronous call ties the caller's fate to the callee's. If the downstream is slow, the caller waits; if it's down, the caller fails; if traffic spikes, the spike hits the callee at full force. A message broker inserts a buffer between them, and that buffer buys three things.

  SYNCHRONOUS                          ASYNCHRONOUS
  producer ──call──▶ consumer          producer ──▶ [ queue ] ──▶ consumer
      │  blocks on consumer                 │  returns now         │  drains at
      │  fails if consumer down             │                      │  its own pace
      ▼                                     ▼                      ▼
  coupled: spike, slowness, and        decoupled: queue absorbs the spike,
  failure all propagate upstream       survives consumer downtime, smooths load

What it buys the upside

Spike absorption — the queue soaks up a burst the consumer couldn't take head-on.
Failure isolation — a down consumer means a growing backlog, not a cascading outage.
Load smoothing — bursty writes become a steady drain at the consumer's comfortable rate.

What it costs the price

Eventual consistency — the work happens later, so reads can race ahead of it.
Harder debugging — no single call stack; you trace a message across systems.
Operational surface — the broker is one more thing to run, size, and monitor.

💡

Async is not free speed — it's a trade. You move latency off the request path and pay with consistency and complexity (Tradeoffs). Use it when the work genuinely doesn't need to finish before you answer the user: sending email, transcoding, updating a search index, fanning out a feed.

📥

Message Queues

work distribution, one consumer per message

A queue is point-to-point work distribution. Producers enqueue tasks; a pool of competing consumers pulls from the queue, and each message is handled by exactly one of them. Add consumers and throughput rises — the queue is a natural load balancer for background work. This is the SQS / RabbitMQ model.

Visibility timeout in-flight protection

When a consumer picks up a message, it's hidden — not deleted — for a timeout window. If the consumer finishes and acks, it's removed. If the consumer dies first, the timeout lapses and the message reappears for someone else. This is how a queue survives worker crashes without losing work.

Dead-letter queue where poison goes

A message that keeps failing would retry forever, blocking the line. After N attempts it's shunted to a dead-letter queue for inspection instead of looping. The DLQ is where you find the poison messages, bad payloads, and downstream bugs.

⚠️

Visibility timeout plus retry means a message can be delivered more than once — a slow consumer's timeout lapses while it's still working, so the message is handed to a second worker. That's the at-least-once reality, and it's why consumers must be idempotent (Concurrency Control).

📡

Pub/Sub

one message, many subscribers

Where a queue delivers each message to one consumer, pub/sub fans out the same message to every subscriber. Producers publish to a topic; each subscriber gets its own copy. This is how one event drives many independent reactions without the producer knowing who's listening.

  QUEUE (one wins)                   PUB/SUB (all get it)
                ┌─▶ consumer A               ┌─▶ email service
  producer ─▶ Q ┤   (only one                │   (own copy)
                └   handles it)   producer ─▶ topic ─┼─▶ analytics service
                                             │   (own copy)
   competing consumers,                      └─▶ search indexer
   load distribution                             (own copy)
                                        fan-out, independent reactions

One "order placed" event can fan out to the email service, the analytics pipeline, and the search indexer at once, each reacting on its own. Subscriptions come in two flavors: ephemeral (you only get messages published while you're connected) and durable (the broker holds your messages until you come back). Durable subscriptions are what make pub/sub reliable across consumer restarts.

🪵

Event Streaming: The Log Model

an append-only log you can replay

Kafka and its kin are a different animal. Instead of deleting a message once consumed, the broker keeps an append-only log, partitioned for scale, retained for days or forever. Consumers track their own offset — a bookmark into the log — so they can replay history, reprocess after a bug, or add a brand-new consumer that reads from the beginning.

  A topic is split into PARTITIONS (the unit of parallelism and ordering):

  partition 0:  [ m0 ][ m1 ][ m2 ][ m3 ][ m4 ]───▶ append
  partition 1:  [ m0 ][ m1 ][ m2 ]────────────────▶ append   order guaranteed
  partition 2:  [ m0 ][ m1 ][ m2 ][ m3 ]──────────▶ append   WITHIN a partition

  Consumer group (each partition → exactly one consumer in the group):
     consumer-A ── reads partition 0        offset: at m3
     consumer-B ── reads partitions 1, 2    offsets tracked per partition

  Parallelism is capped by partition count. More consumers than partitions
  ⇒ the extras sit idle.

Two consequences to remember. Ordering is guaranteed only within a partition, so anything that must stay ordered (all events for one user) has to hash to the same partition key. And partition count bounds parallelism: a consumer group can have at most one consumer per partition doing useful work, so you size partitions for your peak consumer count, not just your data volume (Sharding).

💡

The replayable log is why streaming underpins event sourcing, CDC, and stream processing. The same log feeds a real-time consumer and a batch reprocessing job reading old offsets — one source of truth, many readers at different positions (Batch & Stream, CDC).

🎯

Delivery Guarantees

and why exactly-once is a trick

Three levels, and the interviewer wants to hear that you know the middle one is the real default.

Guarantee	Means	Failure mode
At-most-once	Fire and forget; ack before processing	Messages can be lost on a crash
At-least-once	Ack after processing; redeliver if unsure	Messages can be processed twice
Exactly-once	Every message takes effect once, no more	Expensive; often not what it seems

⚠️

"Exactly-once delivery" over a network is essentially impossible — the sender can never be sure an ack was lost or the message was, so it must choose to risk loss or risk duplication. What's achievable is an exactly-once effect: at-least-once delivery plus an idempotent consumer that ignores duplicates. When someone asks for exactly-once, the answer is "at-least-once with dedup" (Concurrency Control).

🌊

Back-Pressure

when consumers fall behind

A queue absorbs a spike, but an unbounded queue just hides a problem: if consumers are permanently slower than producers, the backlog grows without limit until memory or disk runs out. The buffer bought time, not a solution. Consumer lag — how far behind the newest message the consumers are — is the metric that tells you which is happening.

Response	How it helps
Scale consumers	Add workers (up to the partition limit for logs) to drain faster
Bounded queues	Cap the buffer so the problem surfaces instead of hiding until OOM
Load shedding	Drop or sample low-priority messages under sustained overload
Producer throttling	Slow the producers so they can't outrun consumers indefinitely

ℹ️

Rising lag is an early-warning signal, not a full alarm — the system still works, it's just falling behind. Alerting on lag catches a struggling consumer long before the backlog turns into dropped messages or a full disk (Resilience Patterns).

📋

Quick Reference

queue vs pub/sub vs stream

	Queue	Pub/Sub	Log stream
Delivery	One consumer per message	Every subscriber gets a copy	Every consumer group, by offset
After consume	Message deleted	Delivered, then gone	Retained; replayable
Ordering	Best-effort	Best-effort	Guaranteed within a partition
Replay	No	No	Yes — rewind the offset
Consumer model	Competing workers	Independent subscribers	Consumer groups
Canonical tool	SQS, RabbitMQ	SNS, Google Pub/Sub	Kafka, Pulsar, Kinesis
Best fit	Background jobs, task distribution	Event fan-out to many reactors	Event sourcing, CDC, stream processing

ℹ️

The one-liners to keep ready. Async trades latency-on-the-path for eventual consistency and complexity. Queue = one consumer per message and load distribution; pub/sub = fan-out to many; log = retained, replayable, ordered per partition. Delivery is at-least-once in practice, so make consumers idempotent — that's how you get an exactly-once effect. Watch consumer lag; an unbounded queue only postpones the overload.