Distributed Transactions | System Design Interview Handbook

🔄

The Problem

one action, several stores, no shared transaction

Inside one database, a transaction makes several writes all-or-nothing — ACID handles it (Consistency Models). But "place an order" now touches the order service, the payment service, and the inventory service, each with its own database. There is no transaction that spans them. If payment succeeds and inventory fails, you've charged a customer for something you can't ship.

You can't stretch local ACID across the network. So you reach for one of a few patterns that trade the clean guarantee for something workable.

🤝

Two-Phase Commit

atomic across nodes, at a steep price

2PC keeps strict atomicity with a coordinator and two phases. Prepare: the coordinator asks every participant "can you commit?" and each locks its data and votes yes/no. Commit: if all voted yes, the coordinator tells everyone to commit; if any said no, everyone aborts.

  PHASE 1 — prepare               PHASE 2 — commit
  coordinator ─▶ "prepare?" ─▶    all yes → coordinator ─▶ "commit" ─▶ done
     ├─ payment:   yes (locks)    any no  → coordinator ─▶ "abort"  ─▶ rollback
     ├─ inventory: yes (locks)
     └─ shipping:  yes (locks)    ⚠ participants HOLD LOCKS between phases
                                    ⚠ coordinator dies after prepare?
                                      participants block, locked, waiting

⚠️

2PC is rare in practice for three reasons. Participants hold locks from prepare until commit, killing throughput. It's a blocking protocol: if the coordinator dies after prepare, participants sit locked, unable to decide. And it adds round trips and a single point of failure. It gives real atomicity, but the availability cost is why most systems choose sagas instead.

📜

Sagas

local transactions plus compensation

A saga drops global atomicity and embraces the network. It's a sequence of local transactions, each committing in its own service. If a later step fails, you don't roll back — you run compensating transactions that semantically undo the earlier steps. No locks held across services, so it stays available.

  Payment saga — forward steps, and the compensation path on failure:

  ▶ create order ──▶ charge card ──▶ reserve stock ──▶ ✗ shipping fails
                                                          │
  ◀ cancel order ◀── refund card ◀── release stock ◀──────┘
     (each compensation is a NEW local transaction, not a rollback)

Choreography events, no central brain

Each service listens for events and emits its own. No coordinator — the flow emerges from who reacts to what. Simple for short sagas; hard to follow and debug as steps multiply, because the logic is scattered.

Orchestration a coordinator drives

A central orchestrator tells each service what to do and when, and triggers compensations on failure. One place holds the flow, easier to reason about and monitor; the orchestrator is one more component to build and run.

💡

Compensation is semantic undo, not rollback. You can't un-send an email or un-charge a card — you send a "cancelled" notice or issue a refund. Design each compensating action as a real business operation that reverses the effect, and accept that there's a window where the system is partway done and visibly inconsistent (eventual consistency).

📤

The Outbox Pattern

the default answer to "publish events reliably"

Sagas and event-driven designs assume you can reliably publish an event when you change data. But "commit to the DB" and "publish to the broker" are two systems — you can't make both atomic. Crash between them and you've either updated the DB with no event, or published an event for a change that rolled back. This is the dual-write problem.

  THE TRAP: two writes, no atomicity
     write DB ✓ ──╳ crash ──  publish event ✗   → event lost forever

  THE OUTBOX: one local transaction
     ┌─ BEGIN ────────────────────────────┐
     │  update orders                       │  both commit together,
     │  insert into OUTBOX (the event row)  │  atomically — same DB
     └─ COMMIT ────────────────────────────┘
                    │
         a relay process reads the outbox table
                    ▼
             publishes to the broker, marks the row sent
             (retries safely — at-least-once)

The fix: write the event into an outbox table in the same local transaction as the data change. Now they commit or fail together. A separate relay polls the outbox and publishes to the broker, marking rows as sent. If the relay crashes mid-publish, it retries — so delivery is at-least-once, and consumers must dedup (Concurrency Control). This is the standard, reliable way to turn a state change into an event.

🪝

Change Data Capture

the database as an event source

CDC tails the database's write-ahead log (Debezium is the common tool) and turns every committed change into an event stream (WAL). Since the log records exactly what committed, downstream consumers — search indexes, caches, analytics — stay in sync with zero application code writing events.

ℹ️

Outbox vs raw CDC. Outbox gives you clean, purpose-shaped domain events ("OrderPlaced") but needs the app to write outbox rows. Raw CDC needs no app changes but emits row-level table diffs that consumers must interpret. Outbox when you want well-defined events; CDC when you want to capture everything with no app involvement. Both rest on the same idea: derive the event stream from something already committed, never from a second write.

🔁

Idempotency Is the Glue

every step must tolerate replay

Every pattern here delivers at-least-once: the outbox relay retries, saga steps re-fire, CDC replays after a restart. So every step must be safe to run twice. Idempotency isn't an add-on — it's the property that makes the whole approach correct. A retried "charge card" that charges twice turns a resilience feature into a bug.

💡

Tie it together in an interview: "I'll keep the write and the event atomic with an outbox, relay it to the broker, and coordinate the cross-service flow as a saga with compensations. Since delivery is at-least-once, every consumer dedups on an idempotency key." That sentence covers the whole topic — and idempotency is the load-bearing word (Concurrency Control).

📋

Quick Reference

2PC vs saga vs outbox

	2PC	Saga	Outbox
Consistency	Strong, atomic	Eventual	Eventual (reliable publish)
Latency	High (locks, round trips)	Low per step	Low
Failure behavior	Blocks on coordinator	Compensate backward	Retry the relay
Complexity	Coordinator, locks	Compensations to design	Outbox table + relay
Use when	Rare — true atomicity required	Cross-service business flows	Reliably publishing events

ℹ️

The one-liners to keep ready. Local ACID doesn't cross the network. 2PC gives atomicity but blocks on the coordinator and holds locks — rare. Sagas trade atomicity for availability: local steps plus compensating undo. The dual-write problem is real; the outbox pattern solves it by making the event atomic with the data change, relayed to the broker. CDC derives events from the WAL. Everything is at-least-once, so every step is idempotent.