Think of the write-ahead log as the receipts you keep before updating your bank ledger. Before you erase anything in the ledger, you write a receipt - what the old value was, what the new value will be, and a sequence number. If the power cuts out mid-update and the ledger is corrupt, you pull out the receipts and replay from the last consistent point. The database survives because the log is always written first.

Distributing this concept across multiple nodes transforms a local durability mechanism into a replication protocol. The promise now is not just “survive my own crash” but “survive the crash of any single node in the cluster.” To make that promise, you need consensus: a write isn’t durable until a quorum of nodes have confirmed they’ve appended the entry to their local WAL. Return the acknowledgment before that, and a crash takes your unconfirmed write to the grave.

The design tensions are immediate. Waiting for a quorum before acknowledging writes adds latency - network round trips that stack up under write load. Async replication (acknowledge first, replicate later) gives you low latency but breaks the durability promise: a crash between the ack and replication loses committed data. And recovery requires replaying the log from a checkpoint, which means checkpoints must be consistent, log segments must be retained until they’re no longer needed, and truncation must be coordinated across all nodes.

We need to solve for durability across node failures, tunable replication latency, and point-in-time recovery from any checkpoint simultaneously. This is the engineering challenge that PostgreSQL’s WAL, etcd’s Raft log, and Kafka’s partition log all solve, each with different tradeoffs.

Requirements and Constraints

Functional Requirements

• Append log entries atomically with a monotonically increasing log sequence number (LSN)
• Replicate each entry to at least N/2+1 nodes before acknowledging the write
• Support synchronous and asynchronous replication modes per consumer
• Create checkpoints that allow truncating log segments already applied to state machines
• Recover to any point in time by replaying log from the nearest checkpoint
• Detect and handle split-brain (leader isolation) safely

Non-Functional Requirements

• Write throughput: 100,000 log entries per second per cluster
• Write latency: P99 under 5ms for synchronous mode (LAN replication)
• Log entry size: 128 bytes to 4MB per entry
• Durability: zero committed data loss on any single-node failure
• Recovery time: restore to consistent state within 30 seconds after leader crash
• Replication lag: followers within 1 second of leader under normal load

Constraints and Assumptions

• 3-node and 5-node cluster sizes (odd for quorum clarity)
• Leader-based replication (not multi-master)
• Entries are opaque bytes - the WAL doesn’t interpret log content
• fsync is available and must be called before acknowledging writes
• Network is reliable within a datacenter (LAN latency under 1ms)

High-Level Architecture

The WAL cluster has five key components: the log writer on the leader node that appends entries and calls fsync, the replication engine that streams log entries to followers, the follower log applier on each follower that persists entries and sends acknowledgments, the checkpoint coordinator that manages consistent state snapshots, and the recovery manager that handles leader election and log replay after failures.

A write request flows through the system like this: the client sends an entry to the leader. The leader appends it to its local WAL with the next LSN and calls fsync to guarantee durability on the leader’s disk. Simultaneously, the replication engine streams the entry to all followers. Each follower appends the entry to its local WAL, calls fsync, and sends an acknowledgment back to the leader. The leader waits for N/2 acknowledgments (quorum minus the leader itself), then returns success to the client.

On crash recovery, the new leader - elected from the follower with the highest LSN - replays unconfirmed entries from its log and either commits or aborts them based on whether a quorum holds each entry.

WAL Structure and Log Sequence Numbers

The WAL is the most critical data structure in the system. Its physical layout determines recovery speed, replication efficiency, and disk space utilization.

Think of a WAL as a tape recorder that only plays forward. Entries are written sequentially at the head. The tape never rewinds. When you need to recover, you fast-forward from a known-good position. When a segment of tape is too old to matter, you cut it off.