Scaling Patterns | System Design Interview Handbook

↕️

Vertical vs Horizontal

bigger box, or more boxes

Vertical scaling buys a bigger machine — more CPU, RAM, faster disk. Horizontal scaling adds more machines and spreads load across them. The industry reflex is "scale horizontally," but vertical is underrated early: it's a config change, no distributed-systems complexity, and modern hardware is enormous.

⚠️

Vertical scaling has two hard limits. There's a ceiling — the biggest box money can buy — and a single machine is a single point of failure, no matter how large. Horizontal scaling removes both: near-limitless growth and redundancy built in. So vertical first for simplicity, horizontal when you hit the ceiling or need the redundancy. Both are legitimate; naming which limit you're solving for is the point.

🎈

Statelessness

the property that makes horizontal trivial

Horizontal scaling is easy only if any server can handle any request. That requires statelessness: the server holds no per-user data between requests, so the load balancer can route freely and you can add or kill nodes at will. The instant a server remembers something a client depends on, you've pinned that client to that server and lost the freedom.

  MOVE STATE OFF THE APP SERVER:

     session data ─▶ Redis / signed JWT      (not local memory)
     uploaded files ─▶ object storage         (not local disk)
     local cache ─▶ treated as best-effort     (a miss is fine, never a source of truth)

  result: every app server is interchangeable → add/remove nodes freely,
          the load balancer routes anywhere

The work is pushing state out: sessions into Redis or a signed JWT (Security), files into object storage (Data Stores), and any local cache treated as a disposable optimization, never the truth. Do that and app servers become cattle — interchangeable, replaceable, scalable by the dozen.

🪜

The Scaling Sequence

each step forced by a specific bottleneck

Systems scale in a well-worn order. The value isn't memorizing it — it's naming the bottleneck that forces each step, because that's the reasoning an interviewer grades.

  1. One server (app + DB together)
        │  bottleneck: they compete for CPU/memory
  2. Split the database onto its own machine
        │  bottleneck: one app server maxes out
  3. Load balancer + N stateless app servers
        │  bottleneck: repeated reads hammer the DB
  4. Add a cache layer (Redis in front of the DB)
        │  bottleneck: read QPS still exceeds one DB
  5. Read replicas (spread reads across copies)
        │  bottleneck: slow work blocks request threads
  6. Move heavy work to async queues + workers
        │  bottleneck: one DB can't hold the data / write load
  7. Shard the database

Move	Forced by	See
Separate DB	App and DB contend on one box	—
LB + app servers	One app server saturated	Discovery
Cache layer	Repeated reads on the DB	Caching
Read replicas	Read QPS past one node	Replication
Async queues	Slow work blocking threads	Messaging
Sharding	Write/storage past one node	Sharding

💡

Don't recite all seven steps at the start of a design. Reach for each one when your estimate shows the current design failing on a specific number (Estimation). "Read QPS from my estimate is 50K, past what one Postgres node serves, so I'd add read replicas here" is worth more than listing the whole ladder up front.

🔄

Autoscaling

scale-in is the dangerous direction

Autoscaling adds and removes instances based on a metric — CPU, request rate, queue depth. Two realities make it trickier than "set a target and forget."

Warm-up lag scale-out is slow

A new instance isn't instant — it boots, pulls code, warms caches and connection pools. If you scale out only once you're saturated, the new capacity arrives after the spike hurt. Keep headroom: run above bare minimum so you can absorb a burst while new nodes warm.

Scale-in danger removing capacity

Scaling in too aggressively removes a node just before the next spike, or drops in-flight requests. Scale in slowly and conservatively — it's cheaper to run a little extra than to be caught short. Drain connections before terminating (Real-Time).

🚦

Rate Limiting & Throttling

protecting the system from its clients

Scaling handles legitimate load; rate limiting handles the rest — abusive clients, runaway scripts, retry storms. It caps how many requests a client can make in a window, enforced at the gateway so bad traffic dies before it reaches your services (Discovery). Four algorithms, differing mainly in how they handle bursts.

Algorithm	How it works	Burst behavior
Token bucket	Tokens refill at a steady rate; each request spends one	Allows bursts up to bucket size
Leaky bucket	Requests queue and drain at a fixed rate	Smooths bursts into a steady flow
Fixed window	Count per clock window (per minute)	Spike at window edges (double at the boundary)
Sliding window	Rolling count over the last N seconds	Smooth, no edge spike; more state

💡

Token bucket is the common default — it allows short bursts (good UX) while capping the sustained rate. Enforce at the gateway, return 429 with a Retry-After header so well-behaved clients back off (APIs). This is the heart of the Rate Limiter case study (Rate Limiter).

📋

Quick Reference

bottleneck → move, and rate-limit algorithms

Bottleneck	Move
App + DB contend	Separate the database
App server saturated	Load balancer + stateless app servers
Repeated reads on DB	Cache layer
Read QPS past one node	Read replicas
Slow work blocks threads	Async queues + workers
Write/storage past one node	Shard

Rate limit	Burst	Best for
Token bucket	Allowed, capped	General default, good UX
Leaky bucket	Smoothed away	Steady downstream load
Fixed window	Edge spike	Simple, approximate
Sliding window	Smooth	Accuracy, at more state cost

ℹ️

The one-liners to keep ready. Vertical first for simplicity, horizontal when you hit the ceiling or need redundancy. Statelessness is what makes horizontal easy — push sessions, files, and cache off the app server. Reach for each scaling step only when a number forces it. Autoscale with headroom and scale in cautiously. Rate-limit at the gateway; token bucket is the sane default.