Resilience Patterns | System Design Interview Handbook

🛡

Failure Is the Default

at scale, something is always broken

With enough machines, disks, and network links, something is failing right now. You cannot design failure out — you design to contain it (stop it spreading), degrade gracefully (serve less rather than nothing), and recover automatically. Every pattern on this page is one of those three verbs.

The patterns compose. Timeouts bound how long you wait, retries recover from blips, circuit breakers stop hammering a downed dependency, bulkheads contain the blast radius, and load shedding sheds the least-important work first. Together they break the cascade in the last section.

⏱

Timeouts

the one every remote call needs

Every remote call can hang forever, and a caller waiting forever is a caller holding a thread, a connection, and memory forever. A missing timeout is how "slow" becomes "down" — the slow dependency doesn't just fail its own requests, it exhausts the caller's resources and takes the caller down too.

💡

Propagate deadlines. If the top-level request has 3 seconds, a call three levels deep shouldn't get a fresh 3-second budget — it inherits what's left. Pass the deadline down the chain so nobody works on a request the client already gave up on. "Every remote call has a timeout, and I propagate the deadline" is a cheap, senior line.

🔁

Retries

recover from blips without making a storm

Retries recover from transient failures — a dropped packet, a momentary blip. Two hard rules: retry only transient errors (a 500, a timeout — never a 400), and only idempotent operations, or a retry double-charges the customer (Concurrency Control).

  Naive retry (all clients retry at the same instant):
     fail ─▶ retry ─▶ retry ─▶ retry     synchronized WAVES hammer the
      │       │        │        │        recovering service back down
      └───────┴────────┴────────┘

  Exponential backoff + JITTER:
     wait 1s ± rand ─▶ wait 2s ± rand ─▶ wait 4s ± rand
     spacing grows, randomness spreads clients out → no thundering wave

⚠️

Retries are dangerous precisely when a service is already struggling — that's when everyone retries at once. Exponential backoff spaces attempts out; jitter (randomness) desynchronizes clients so they don't retry in lockstep. And beware retry amplification: if every layer retries 3×, three stacked layers turn one request into 27 against the bottom service. Use a retry budget — cap retries as a fraction of traffic — and retry at one layer, not every layer.

🔌

Circuit Breaker

stop hammering a dependency that's down

When a dependency is clearly down, retrying is worse than failing fast — you waste your own resources and pile load on something that can't answer. A circuit breaker watches the failure rate and, once it crosses a threshold, opens: calls fail immediately without even trying, giving the dependency room to recover.

            failures exceed threshold
   ┌────────┐ ───────────────────────▶ ┌────────┐
   │ CLOSED │                           │  OPEN  │  fail fast,
   │ normal │ ◀─────────────────────    │ no calls│  don't even try
   └────────┘   probe succeeded         └────┬───┘
        ▲                                    │ after a cooldown
        │       ┌───────────┐                │
        └────── │ HALF-OPEN │ ◀──────────────┘
       probe    │ let a few  │
       fails →  │ test calls │  probe passes → close again
       reopen   │ through    │
                └───────────┘

After a cooldown it goes half-open, letting a few probe requests through. If they succeed, it closes and normal traffic resumes; if they fail, it opens again. The breaker protects both sides: the caller stops wasting time on doomed calls, and the struggling callee gets breathing room instead of a stampede.

🚢

Bulkheads

isolate resources so one leak doesn't sink the ship

Named after a ship's watertight compartments: a breach floods one, not the whole hull. In software, you give each dependency its own isolated pool of resources — connections, threads, instances — so one slow dependency can only exhaust its pool, not drain every thread in the service.

  WITHOUT bulkheads: one shared thread pool
     service A slow ─▶ all threads block on A ─▶ calls to healthy B, C also starve

  WITH bulkheads: a pool per dependency
     ┌── pool A ──┐  A is slow → only pool A fills
     ┌── pool B ──┐  B still served
     ┌── pool C ──┐  C still served     the failure is contained to A

ℹ️

Without isolation, a single slow dependency is enough to hang an entire service — every incoming request eventually blocks on the one slow call and the shared pool drains. Bulkheads cap that damage to the one dependency's share. This is the containment step that stops the cascade below.

📉

Load Shedding & Graceful Degradation

serve less rather than fail entirely

Under overload, the choice is not "serve everyone" — it's "serve some, or serve no one." Load shedding drops low-priority work so high-priority work survives: reject non-critical requests, turn off expensive features, return early. Graceful degradation serves a worse-but-real answer — stale cache, partial results, a default recommendation — instead of an error.

💡

Feature flags as kill switches make this operational: when the recommendation service is melting, flip a flag to serve a generic list and keep the page loading. The product degrades a little; it doesn't go down. "Under load I'd shed the personalization and serve a cached default rather than fail the whole request" is exactly the degradation instinct interviewers want to hear.

🌩

Cascading Failure Anatomy

how one slow DB takes down everything

Cascading failure is how a small problem becomes a total outage. Walk one chain, then see how each pattern above breaks a link.

  1. DB slows down (a bad query, a lock, a spike)
        │                                    ← TIMEOUT bounds the wait
  2. app threads block waiting on the DB
        │                                    ← BULKHEAD isolates the DB pool
  3. thread pool exhausts; healthy requests can't get a thread
        │                                    ← LOAD SHEDDING drops low-priority work
  4. health checks time out → node marked unhealthy, pulled from the LB
        │                                    ← shallow health check (Discovery)
  5. its traffic reshuffles onto the remaining nodes
        │                                    ← CIRCUIT BREAKER stops the DB pile-on
  6. now THEY are overloaded → they tip over → total outage
                                             ← BACKOFF + JITTER prevents retry waves

⚠️

The vicious part is step 5: removing an overloaded node moves its load onto the survivors, pushing them over the same edge — the failure accelerates as it spreads. A deep health check makes it worse by taking every node out at once when the shared DB blips (Discovery). The patterns aren't academic; each one severs a specific link in this exact chain.

📋

Quick Reference

patterns on one card

Pattern	Protects against	Key parameter	Gotcha
Timeout	Hung calls exhausting resources	Deadline (propagated)	Missing one turns slow into down
Retry	Transient blips	Backoff, jitter, budget	Amplifies load if stacked
Circuit breaker	Hammering a downed dependency	Failure threshold, cooldown	Tune so it doesn't flap
Bulkhead	One dependency draining all resources	Pool size per dependency	Over-partitioning wastes capacity
Load shedding	Overload collapsing everything	Priority classes	Must know what's low-priority
Graceful degradation	All-or-nothing failure	Fallback path	Fallback must be truly cheap

ℹ️

The one-liners to keep ready. Failure is the default; contain, degrade, recover. Every remote call gets a timeout with a propagated deadline. Retry only transient, idempotent calls, with backoff and jitter, at one layer. Circuit breakers fail fast to protect both sides. Bulkheads isolate the blast radius. Shed low-priority load and degrade before you fail. Each pattern cuts a specific link in the cascade chain.