Chapter 8 is the pessimist's chapter. After several chapters that assumed things mostly work, Kleppmann turns to everything that can go wrong in a distributed system — and the list is long and humbling. The defining feature is partial failure: some parts of the system work while others are broken, in nondeterministic ways you often can't even detect. A single computer is deterministic (it works or it crashes cleanly); a distributed system can be in a maddening in-between state. The chapter's goal is to build an accurate mental model of these faults — unreliable networks, unreliable clocks, and process pauses — so that the consensus tools of Chapter 9 make sense.
- Partial failure is the essence — distributed systems fail in nondeterministic, partial ways a single machine never does. You must assume anything that can break, will.
- Networks are unreliable — a lost request, a dead node, and a slow response are indistinguishable from the sender's side. The only tool is a timeout, and there is no perfect timeout.
- Clocks are unreliable — time-of-day clocks can jump (NTP, leap seconds), so ordering events across nodes by wall-clock timestamp (e.g. LWW) is unsafe.
- Processes pause unpredictably — GC, VM suspension, and OS scheduling can freeze a node for seconds; a "leader" may pause past its lease and not know it.
- Fencing tokens defend shared resources — a monotonically increasing token lets storage reject a stale node that woke up thinking it still holds the lock.
- A node can't trust itself — truth is defined by a quorum (majority), not by any single node's opinion. Byzantine faults (lying nodes) are a harder, usually datacenter-irrelevant case.
Distributed systems are built on three fundamentally unreliable foundations — the network, clocks, and the timely execution of code — and none of them can be trusted to behave within bounds. The practical consequences: detect failures only via timeouts (imperfectly), never order events across nodes by wall-clock time, defend shared resources with fencing tokens, and let a majority quorum — not any single node — decide what's true.
Faults and Partial Failures
On a single computer, operations are deterministic: either the whole machine works, or it crashes. We deliberately prefer a computer to crash entirely rather than return a wrong result. Distributed systems abandon this comfort. With many machines connected by a network, there will be partial failures: parts work while others are broken, and crucially the failure is nondeterministic — an operation involving the network may unpredictably succeed, fail, or hang, and you may not even know which. This nondeterminism, not any single fault, is what makes distributed systems hard. The cloud-computing approach (commodity machines, expect failure, build fault tolerance in software) differs from the supercomputing approach (treat the whole cluster like one machine, restart on failure). For internet services we want continuous availability, so we must build a reliable system from unreliable components.
Unreliable Networks
The distributed systems in this book are shared-nothing: machines communicate only by passing messages over an asynchronous packet network. When you send a request and don't get a response, you cannot tell what happened. Many possibilities collapse into the same observation:
- Your request was lost (cable unplugged, switch overloaded).
- Your request is queued and will be delivered later.
- The remote node failed (crashed or powered down).
- The remote node is temporarily paused and will respond later.
- The remote node processed your request, but the response was lost or delayed.
From the sender's side, all of these look identical: no reply. Network faults are not rare even inside a single datacenter — studies cited in the book find them common. The only practical way to handle an unresponsive node is a timeout: stop waiting after some time and assume failure.
Timeouts and Unbounded Delays
But how long is the right timeout? There's no good answer. A long timeout means a long wait before declaring a dead node dead (slow recovery). A short timeout detects faults faster but risks false positives — declaring a node dead when it's merely slow, which can be catastrophic: the node's work gets handed to others, increasing their load, possibly making them time out, cascading into a death spiral. Delays are unbounded because packet networks use statistical multiplexing: queueing at switches, at the destination's OS, and from other traffic all add variable delay. In contrast, a traditional circuit-switched telephone network reserves bandwidth and offers bounded delay — but at the cost of poor utilization for bursty data. You can't have both high utilization and bounded latency, so the right move is to measure round-trip times continuously and adapt timeouts rather than hard-coding one.
Unreliable Clocks
Time seems simple but is treacherous in distributed systems, because each machine has its own clock (a quartz oscillator that drifts), synchronized imperfectly over the network with NTP. Two kinds of clocks must not be confused:
| Aspect | Time-of-day clock | Monotonic clock |
|---|---|---|
| Measures | Wall-clock time (e.g. UTC) | Elapsed time since some point |
| Can jump backward? | Yes — NTP, leap seconds | No — always moves forward |
| Comparable across nodes? | Supposed to be, but unreliable | No — meaningless across nodes |
| Use it for | Timestamps, calendars | Measuring durations/timeouts |
The Danger of Trusting Synchronized Clocks
Clock synchronization is much less reliable than people assume: NTP is limited by network round-trip delay, nodes drift, and leap seconds and misconfigurations cause clocks to jump. The most dangerous consequence is using wall-clock timestamps to order events across nodes — for example, last-write-wins (LWW) conflict resolution. If node A's clock is even slightly ahead of node B's, a write from B that genuinely happened later can be silently discarded because its timestamp looks older. Clock skew thus causes data loss and causality violations that are nearly impossible to debug. If you must use clocks for ordering, you need confidence intervals: a timestamp is really a range, not a point. Google's Spanner uses TrueTime (GPS and atomic clocks) to bound the uncertainty, and deliberately waits out the uncertainty interval before committing, so the ordering is guaranteed.
Process Pauses
Even setting clocks aside, you cannot assume code runs in a timely way. A thread can be paused at any point for an arbitrarily long time: a garbage collection stop-the-world pause, a virtual machine being suspended and migrated, the OS preempting the thread, the laptop lid closing, or even slow disk I/O. The classic failure: a node holds a lease saying "I am the leader until time T," does some work, but pauses (say a 15-second GC pause) right before acting. By the time it resumes, the lease has expired, another leader has taken over — yet the paused node doesn't know and proceeds as if it's still in charge, corrupting shared state.
Knowledge, Truth, and Lies
The recurring lesson is that a node cannot trust its own judgment about the state of the system — it only knows what messages it has (or hasn't) received, and those can be lost or delayed. So distributed systems rely on a quorum: decisions require agreement among a majority of nodes, so the system doesn't depend on any single node. A node may believe it's the leader, but if the quorum has moved on, its belief is irrelevant — and dangerous.
Fencing Tokens
To protect a shared resource from the paused-leader problem, use a fencing token: every time the lock/lease is granted, the lock service also returns a number that increases every time. The client must include this token with every write to the protected resource, and the resource rejects any write carrying a token older than one it has already seen. So when a paused old leader wakes up and tries to write with a stale token, the storage rejects it.
client A gets the lock → token 33
client A pauses (long GC) …
lease expires; client B gets the lock → token 34
client B writes(token=34) → storage accepts, records 34
client A wakes, writes(token=33) → storage sees 33 < 34 → REJECTED
the storage service enforces the order — clients can't be trusted
to know they've been superseded.Byzantine Faults
Everything above assumed nodes are honest but may be slow, crash, or lose messages. A Byzantine fault is worse: a node that behaves arbitrarily — sending corrupted or contradictory messages, perhaps maliciously (the name comes from the "Byzantine Generals Problem"). Byzantine fault-tolerant algorithms keep working even when some nodes lie, but they're complex and expensive. In a typical trusted datacenter you can assume no Byzantine faults (the nodes are yours), so most systems don't pay this cost. They matter in adversarial settings: aerospace (radiation flipping bits), and peer-to-peer/blockchain systems where participants don't trust each other.
System Models
To reason rigorously, we define a system model — assumptions about timing (synchronous: bounded delays; partially synchronous: usually bounded but occasionally not — the realistic one; asynchronous: no timing assumptions) and about node failures (crash-stop, crash-recovery, Byzantine). Correctness is expressed as safety properties ("nothing bad happens" — must always hold) and liveness properties ("something good eventually happens" — may have caveats like "eventually"). Good algorithms prove their safety properties hold in all the runs the model permits.
This chapter is deliberately bleak so the next one can be constructive. The single most useful habit it instills: stop assuming the network is reliable, the clock is accurate, or your code runs on time. Design as if requests will be lost, timestamps will lie, and your process will freeze at the worst moment — then lean on quorums and fencing tokens instead of any node's local belief.
Why can't you tell a dead node from a slow one? A shared-nothing node only observes "no reply," which could be a lost request, a crashed peer, or a slow response — so timeouts are the only (imperfect) detector.
Why is LWW with wall-clock timestamps unsafe? Clock skew between nodes means a genuinely later write can carry an "older" timestamp and be silently dropped — data loss from causality violation.
What problem do fencing tokens solve? A paused leader waking after its lease expired; a monotonically increasing token lets the storage reject the stale writer.
Safety vs liveness? Safety = "nothing bad happens" and must always hold; liveness = "something good eventually happens" and may be qualified by "eventually."