Metrics, Logs & Traces | System Design Interview Handbook

📊

Three Signals, Three Questions

each answers a different question

Observability is three tools, and the mistake is treating them as interchangeable. Each answers one question, and the art is escalating from cheap to expensive: a metric tells you something's wrong, a trace tells you where, and a log tells you exactly what.

  METRICS ─▶ "Is something wrong?"     cheap, aggregated numbers, alertable
      │                                 (error rate spiked)
      ▼  something's wrong — where?
  TRACES  ─▶ "Where in the chain?"     one request across services
      │                                 (the payment call is timing out)
      ▼  which call — what happened?
  LOGS    ─▶ "What exactly happened?"  rich, detailed, expensive, searchable
                                        (stack trace, the bad input)

ℹ️

The escalation is the whole point. You can't afford to log everything at metric volume, and you can't alert on individual logs. Metrics catch the problem, traces localize it, logs explain it. Reaching for the right one at the right moment is what separates a fast incident from a three-hour one.

📈

Metrics

the golden signals, and why averages lie

Metrics are cheap aggregated numbers over time — counters (requests served), gauges (current connections), histograms (latency distribution). The framework to name is the four golden signals: latency, traffic, errors, saturation. Instrument those per service and you can see health at a glance.

  WHY PERCENTILES, NOT AVERAGES:
     1000 requests. 990 at 50ms, 10 at 5000ms.
        average = 99ms   ← looks fine, hides everything
        p99     = 5000ms  ← the truth: 1 in 100 users waited 5 seconds

  average smears the outliers away; p99 / p999 expose the tail users feel.

⚠️

Two traps. Averages lie — always alert and reason on percentiles (p99, p999), because the tail is where real users suffer and the mean hides it. And cardinality kills metrics systems: tagging a metric with a high-uniqueness label like user ID explodes the number of series stored and can take the whole metrics pipeline down. Keep labels low-cardinality; user-specific detail belongs in logs, not metric tags.

📜

Logs

rich detail, expensive at volume

Logs are the detailed record of what actually happened — the stack trace, the offending input, the exact sequence. Rich but expensive, so a few practices keep them useful at scale.

Practice	Why
Structured logging	JSON fields, not free text — so you can query and aggregate, not grep
Centralized aggregation	Ship logs off the box to one searchable place; a dead server's logs survive
Sampling under volume	Keep all errors, sample the routine — full logging at scale is unaffordable
Correlation IDs	One ID threaded through a request ties its logs together across services

💡

The correlation ID is the load-bearing detail. Generate one at the edge, pass it through every service and queue, and stamp it on every log line. Now "show me everything that happened to this one failing request" is a single query across the whole system — otherwise you're stitching timestamps by hand.

🧵

Distributed Tracing

where the time went, across services

In a microservices system, one user request fans out across many services, and "which one was slow?" isn't answerable from any single service's view. A trace stitches the whole path together: each unit of work is a span, spans nest into a tree, and the trace shows where every millisecond went.

  ONE TRACE (one request):
   gateway  ├────────────────────────────────────────┤  120ms
   auth       ├───┤                                       8ms
   orders          ├──────────────────────┤              70ms
   payment              ├───────────────┤                55ms  ← the culprit
   db                     ├──┤                            9ms

  each bar is a SPAN; trace context is propagated so spans link into one tree

The key mechanism is context propagation: the trace ID travels with the request across service and queue boundaries, so an async job triggered by a message still links back to the request that caused it. OpenTelemetry is the standard for this. Because tracing every request is costly, you sample — head-based (decide at the start) or tail-based (decide after, keeping the slow and failed ones, which are the interesting ones).

🎤

In the Interview

cheap points at wrap-up

Observability is rarely the focus of a design question, which is exactly why one crisp sentence at the end scores easy points few candidates claim. It signals you think about running the system, not just building it.

💡

The wrap-up line: "I'd instrument the four golden signals per service, alert on the SLO breach not on CPU, and propagate a trace context across the queue boundary so I can follow a request end to end." Metrics for detection, traces for localization, logs with correlation IDs for the detail. Tie the alert to the user-facing SLO and you've connected this page to the next one (SLI/SLO/SLA).

📋

Quick Reference

the three signals compared

	Metrics	Logs	Traces
Answers	Is something wrong?	What exactly happened?	Where in the chain?
Cost	Cheap	Expensive at volume	Moderate, sampled
Cardinality	Must stay low	High is fine	Per-request
Alertable?	Yes — the primary signal	Rarely directly	No
Scope	Aggregated over time	One event	One request, cross-service

ℹ️

The one-liners to keep ready. Three signals, three questions: metrics (wrong?), traces (where?), logs (what?). Instrument the four golden signals — latency, traffic, errors, saturation. Alert on percentiles, never averages, and keep metric cardinality low. Thread a correlation ID through logs and a trace context across service and queue boundaries. OpenTelemetry is the standard; sample traces, keeping the slow and failed ones.