Load Balancing & Discovery | System Design Interview Handbook

⚖️

Load Balancing Algorithms

how to pick which instance gets the request

Given a pool of healthy instances, the load balancer picks one per request. The algorithm choice matters when requests aren't uniform — some are slow, some instances are bigger, some clients benefit from landing on the same node twice.

Algorithm	How it works	Best for
Round robin	Next instance in rotation	Uniform requests, equal instances
Weighted	Round robin biased by capacity	Mixed instance sizes
Least connections	Fewest active connections wins	Long-lived or uneven-duration requests
Least response time	Fastest-responding instance wins	Latency-sensitive, variable backends
IP / consistent hash	Hash the key to a stable instance	Session affinity, cache locality

💡

Round robin is the safe default. Reach for least connections when request durations vary a lot — round robin can pile short and long requests unevenly. Reach for consistent hashing when landing on the same instance twice is valuable, like keeping a user's data in one node's local cache (Sharding).

🧅

L4 vs L7

how deep the balancer looks

A load balancer can operate at two layers. L4 works at the transport level — it sees IPs and ports, forwards packets, and never opens the payload. L7 works at the application level — it terminates the connection, reads the HTTP request, and can route on path, header, or cookie.

  L4 (transport)                        L7 (application)
  sees: IP : port                       sees: full HTTP request
  ┌──────────────┐                      ┌──────────────────────────┐
  │  forward the │  ──▶ backend         │  GET /api/*  ──▶ api pool  │
  │  TCP packets │      (blind to        │  GET /img/*  ──▶ cdn pool  │
  │  fast, cheap │       content)        │  header: beta ──▶ canary   │
  └──────────────┘                      └──────────────────────────┘
   can't route by URL,                   routing, TLS termination,
   can't retry a request                 retries, rewrites — at a cost

	L4	L7
Sees	IP, port, TCP	Full HTTP: path, headers, cookies
Can do	Fast forwarding, any protocol	Path routing, TLS termination, retries, rewrites
Cost	Minimal overhead	Terminates and parses — more CPU
Use when	Raw throughput, non-HTTP traffic	Smart routing on HTTP semantics

ℹ️

Most web systems use L7 at the edge because routing by path and terminating TLS there is worth the CPU (Networking Basics). L4 shows up where you need raw speed or non-HTTP protocols, or as a first hop in front of an L7 tier.

🩺

Health Checks

knowing which instances can take traffic

The balancer only routes to instances it believes are healthy, so it constantly probes them. Active checks send a periodic request and watch for a good response. Passive checks infer health from real traffic — a burst of errors or timeouts marks an instance down without a separate probe.

Checks also vary in depth. A shallow check confirms the process is up ("return 200 on /health"). A deep check verifies dependencies too — can it reach the database, the cache, the downstream service.

⚠️

Deep health checks are a double-edged tool. If every instance's health check pings a shared database and that database hiccups, every instance reports unhealthy at once — and the balancer takes the entire fleet out of rotation over a blip. A shared dependency failing should degrade service, not vanish the whole pool. Keep the check shallow, or fail it only on sustained, local problems (Resilience Patterns).

🛰

Service Discovery

finding instances that come and go

In a dynamic fleet, instances start, stop, crash, and autoscale constantly. Nobody can hard-code addresses. A service registry is the live directory: instances register on startup and send heartbeats; they deregister on shutdown, or the registry evicts them when heartbeats stop. Callers ask the registry where a service lives.

Client-side discovery caller picks

The client queries the registry, gets the list of instances, and load-balances itself. Fewer hops and full control, but every client needs the discovery logic and a fresh view of the registry.

Server-side discovery balancer picks

The client just calls a stable load balancer, which consults the registry and forwards. Clients stay dumb; the routing tier owns discovery. One more hop, one more thing to run.

⚠️

The registry can be stale in both directions. It may still list an instance that just died (calls fail until it's evicted) or not yet list one that just started (traffic misses it). Short heartbeat intervals and retries on the caller side paper over the gap. The registry itself needs strong consistency for membership, which is why it's usually built on a consensus store (Consensus).

🚪

API Gateway

edge concerns in one place

An API gateway is the single front door to a system of services. It handles the cross-cutting edge concerns you don't want scattered across every service: authentication, rate limiting, routing, request shaping, and response aggregation. One place to enforce policy, one place clients target.

   clients
      │
      ▼
  ┌─────────────────────────────────────────┐
  │            API GATEWAY                    │
  │  authn · rate limit · routing · shaping   │
  └───┬───────────┬───────────┬───────────────┘
      ▼           ▼           ▼
   user svc    order svc    search svc     (internal services stay focused
                                            on business logic, not edge policy)

The terms blur on purpose. A reverse proxy forwards requests to backends. A load balancer spreads them across instances. An API gateway adds application-aware policy on top. In practice one component often does all three, which is why the words get used interchangeably — the distinction is the responsibility, not the box (Security Basics, Rate Limiter).

🌍

Global Load Balancing

routing users across regions

Everything above balances within one region. Across regions, two mechanisms send a user to the right one. GeoDNS answers a DNS query with the nearest region's address based on the resolver's location. Anycast advertises one IP from many locations, and the network routes packets to the closest — routing at the network layer, so it reacts faster than DNS and needs no TTL wait.

ℹ️

GeoDNS is simple but coarse and slow to fail over — it sees the resolver, not the user, and is bound by TTLs (Networking Basics). Anycast is more precise and reroutes near-instantly when a location goes dark, which is why CDNs and DNS providers run on it. Getting the user to the right region is only half the job; what happens to their data across regions is a separate, harder problem (Multi-Region).

📋

Quick Reference

algorithms and layers on one card

Algorithm	How it works	Best for
Round robin	Rotate through instances	Uniform requests — the default
Weighted	Biased by capacity	Mixed instance sizes
Least connections	Fewest active wins	Variable request durations
Least response time	Fastest wins	Latency-sensitive backends
Consistent hash	Key → stable instance	Session affinity, cache locality

	L4	L7
Sees	IP, port	Full HTTP
Can do	Fast forwarding	Path routing, TLS term, retries
Use when	Throughput, any protocol	Smart HTTP routing

ℹ️

The one-liners to keep ready. Round robin by default; least-connections for uneven requests; consistent hash for affinity. L7 at the edge for routing and TLS, L4 for raw speed. Keep health checks shallow so a shared dependency doesn't take the whole fleet down. A registry with heartbeats handles a fleet that's always changing, and it's stale in both directions — retry. Gateway, load balancer, and reverse proxy are three responsibilities that often live in one box.