Imagine 50 workers at a construction site sharing one excavator. Without a checkout board, three workers grab the controls simultaneously, each convinced they have exclusive access. The machine fights itself. At best, they waste time undoing each other’s work. At worst, someone gets hurt. A lock manager is that checkout board for distributed systems - a shared authority that ensures only one process controls a shared resource at any instant.

The checkout-board analogy breaks down at scale. A physical board is a single point of failure: if the site supervisor goes home, nobody can check out equipment. A distributed lock manager must itself be highly available. It needs to survive the crash of any individual node while maintaining the invariant that at most one holder can possess each lock. That tension - availability versus mutual exclusion - is what makes building this service hard. Zookeeper solved it with ZAB consensus in 2010, etcd with Raft in 2013, and Google’s Chubby with Paxos internally. Each of these systems exists because the naive alternatives all fail in production.

Three failures lurk in naive implementations. First, a lock holder that crashes while holding the lock blocks every other waiter indefinitely - unless the lock has a time-to-live that expires it automatically. Second, a lock holder that takes too long due to garbage collection or network partition can have its lease expired by the lock service while it still believes it holds the lock, then both the original holder and the new acquirer are writing to the shared resource simultaneously - the classic split-brain scenario. Third, when the lock service itself fails over to a new leader, waiters that rush to acquire newly freed locks all arrive at once - the thundering herd that can overwhelm the recovering service.

Solving these cleanly requires three core architectural decisions: Raft consensus to make the lock service itself highly available without split-brain, lease-based locking with fencing tokens to protect resources from stale lock holders, and a watch API so clients react to leadership changes in near-real-time without polling.

Requirements and Constraints

Functional Requirements

• Provide mutual exclusion: at most one client holds a named lock at any instant
• Support lease-based locks with configurable TTL (1s to 600s); expired automatically on holder failure
• Support leader election: exactly one candidate wins the “leader” lock per named election
• Provide a watch API: clients subscribe to lock/leader change events with at-most-once delivery
• Issue monotonically increasing fencing tokens with each lock grant to protect downstream resources
• Support lock hierarchy: namespaced keys (e.g., /locks/billing/batch-job)
• Allow lock renewal before expiry (heartbeat-based keepalive)

Non-Functional Requirements

• Lock service availability: 99.99% (four nines) with automatic failover under 500ms
• Lock acquisition latency: p50 under 5ms, p99 under 20ms within a datacenter
• Support 50,000 concurrent lock holders per cluster
• Support 10,000 lock acquire/release operations per second per shard
• Watch event delivery latency: under 100ms after state change
• Fencing token counter must never decrease across leader elections

Constraints and Assumptions

• Minimum cluster size is 3 nodes for a quorum majority of 2; production uses 5 nodes for tolerance of 2 simultaneous failures
• Network round-trip time between nodes is under 5ms; design does not address WAN consensus
• Clients run their own heartbeat loops; the server does not push keep-alive reminders
• Lock service is a coordination primitive, not a general-purpose key-value store - maximum value size is 4KB

High-Level Architecture

Six components make the system work: a gRPC API layer that clients talk to for lock acquire/release/watch, a Raft consensus cluster that replicates all state changes across nodes, a Lease Store that holds the in-memory state of every active lock with its TTL and fencing token, a Fencing Token Counter (a monotonically increasing integer per lock key, replicated through Raft), a Watch Engine that maintains client subscriptions and pushes state-change events, and a Lease Reaper that continuously expires TTL-exceeded leases.

Data flows like this: a client sends AcquireLock(key="billing/job", ttl=30s) to any cluster node. Followers proxy write requests to the current Raft leader. The leader proposes the lock grant as a Raft log entry - {op: LOCK, key: "billing/job", holder: "client-A", token: 42, expires_at: T+30s}. Once a majority of nodes append this entry to their log and commit it, the leader applies the entry to its in-memory Lease Store, increments the fencing token counter, and responds to the client with {granted: true, token: 42, expires_at: T+30s}. The Watch Engine simultaneously notifies all subscribers watching billing/job that it has been acquired.

When the client’s TTL is about to expire, it sends RenewLock(key, token=42) to extend the lease. If the client crashes before renewing, the Lease Reaper running every 100ms detects the expiry, Raft-commits a {op: DELETE, key: "billing/job"} entry, and the Watch Engine notifies all waiters that the lock is free. The next acquirer gets token=43 - and any write the crashed client attempts using token=42 is rejected by the downstream resource.

The Lock Service

The Lock Service is the front door for all client operations. It wraps a Raft state machine that ensures every lock grant, renewal, and release is replicated to a majority of cluster nodes before acknowledging the client. Any node can serve read-only requests (checking whether a key is locked); only the Raft leader can process writes.

Think of the Raft cluster like a parliamentary vote: a motion is not carried until a majority of members raise their hands. The leader proposes the motion (lock grant), collects acknowledgments from followers, and only then announces the result. If the leader’s connection to the parliament drops mid-vote, the parliament elects a new speaker and starts fresh - but no motion is ever announced without a recorded majority.

The critical property is that the fencing token counter must survive leader elections with strictly increasing values. We achieve this by storing the counter in the Raft log itself. When a new leader is elected, it replays the log from the last committed entry, reconstructing the exact counter value where the previous leader left off. No token is ever reused.