Picture a bank account shared by five siblings, each holding their own debit card, all drawing against the exact same balance. If two siblings check the balance at the same instant, both see $200, and each walks up to a different ATM three blocks apart and withdraws $150, the bank has no way of knowing about the overdraft until both withdrawals have already been dispensed. The account was never actually short of money at either individual moment of checking, it only became short once both withdrawals landed, and by then the cash is already in someone’s pocket.

Now replace the five siblings with fifty regional warehouses, replace the two ATMs with a checkout button that gets clicked by thousands of shoppers a second, and replace the shared bank balance with the last unit of a doorbuster item during a flash sale. That is the oversell problem: a piece of physical stock that exists in exactly one place can be shown as “available” to two different shoppers at once if the system checking availability and the system recording a sale are not the same atomic operation. Selling something you do not have is not a rounding error, it is a promise the business cannot keep, and the fix a customer support team offers afterward (a refund, a discount, an apology email) costs far more than the sale was worth.

The naive fix, decrement the count the moment payment succeeds, sounds safe but creates its own failure: a shopper adds the last unit to their cart, gets distracted by a slow card entry form, and thirty seconds later a second shopper across the country sees the item as “in stock,” because nothing was held for the first shopper while they typed. Decrementing too late lets two people believe they bought the same unit. Decrementing too early, the moment an item is added to a cart with no expiration, creates the opposite failure: an abandoned cart holds inventory hostage forever, and a warehouse can show “sold out” on an item that is, in physical reality, sitting untouched on a shelf. Both mistakes come from treating “browsing,” “checking out,” and “paying” as if they happen instantly and atomically, when in reality a real checkout session can take anywhere from three seconds to several minutes, and across that window somebody else is always trying to buy the same thing.

We need to solve for three things simultaneously: an atomic counter that fifty warehouses can update concurrently without ever going negative, a time-bound soft hold that reserves stock for exactly as long as a checkout session needs and no longer, and a routing layer that can satisfy an order from whichever of the fifty warehouses actually has the stock, even when that means splitting the order across more than one.

Requirements and Constraints

Functional Requirements

• Reserve a requested quantity of a SKU against warehouse stock the moment a shopper begins checkout, attaching a TTL (default 15 minutes) to that reservation
• Commit a reservation into a permanent sale the instant a payment webhook confirms success, converting the soft hold into a durable decrement of on-hand stock
• Release a reservation automatically, without any client action, the moment its TTL expires or the shopper explicitly abandons checkout
• Route a reservation request across whichever warehouse or combination of warehouses in the network can fulfill it, splitting into multiple shipments when no single warehouse holds the full requested quantity
• Support both an optimistic, retry-based concurrency strategy and a pessimistic, lock-based strategy, and pick between them per SKU based on how contended that SKU currently is
• Emit a sold-out signal to the product catalog and search layer the instant a SKU’s network-wide available count reaches zero, and a back-in-stock signal once it recovers
• Guarantee that every reserve, commit, and release API call is idempotent, so a client’s network retry never creates a duplicate hold or a duplicate release

Non-Functional Requirements

• Network scale: 50 regional warehouses, roughly 2,000,000 distinct SKUs, on the order of 500,000,000 total units on hand across the network
• Peak reservation throughput: sustained 15,000 reservation requests/sec in normal operation, bursting to 80,000 requests/sec during flash sales and major promotional windows
• Latency: reserve p99 under 50ms, commit and release p99 under 100ms
• Correctness guarantee: the available count for any warehouse/SKU pair must never go negative under any level of concurrency; this is a harder requirement than latency or throughput
• TTL: default hold duration of 15 minutes, configurable per campaign, with expired holds released back to the available pool within 5 seconds of expiry
• Availability: the reserve path, being directly on the revenue path, targets 99.99% uptime; fulfillment routing is allowed to degrade to a single-warehouse fallback rather than take the whole reserve path down
• Consistency scope: a reservation is scoped to a single request’s set of warehouses; there is no requirement to keep two independent reservations for the same SKU strictly ordered relative to each other beyond the correctness guarantee above

Constraints and Assumptions

• A separate catalog and pricing system owns product metadata; this system only owns quantity counts and reservation state, not price or description
• The actual physical pick, pack, and ship process is owned by warehouse operations systems; this system’s job ends once a reservation is committed into a sale
• The payment gateway’s webhook is treated as the source of truth for “did the customer actually pay,” we do not design payment processing itself
• Warehouses report physical stock counts (receiving, cycle counts, returns) into this system through a separate replenishment feed, which we treat as an upstream input, not something this design owns
• Cross-region inventory sharing (routing an order from a customer in one country to a warehouse in another) is out of scope for this design; all 50 warehouses are assumed to sit inside one logical fulfillment region

High-Level Architecture

The system has six major components: the Reservation Gateway, the Distributed Inventory Counter Store, the Fulfillment Routing Engine, the Reservation Ledger, the TTL Expiry Sweeper, and the Sold-Out Signal Bus.

The Reservation Gateway is the front door: it validates a reservation request, deduplicates retries using an idempotency key, and hands the request to the routing layer. The Fulfillment Routing Engine decides which warehouse, or which combination of warehouses, should actually supply the requested quantity, scoring candidates by distance, cost, and available stock. The Distributed Inventory Counter Store holds the authoritative available/reserved/on-hand counts per warehouse per SKU and is the only component allowed to mutate a count, using an atomic operation so two concurrent requests can never both succeed against the same last unit. The Reservation Ledger is the durable system of record for every hold, tracking its state through held, committed, released, and expired, and is what a reconciliation job or a customer support agent can query after the fact. The TTL Expiry Sweeper actively finds holds whose TTL has lapsed and releases their stock back to the counter store, rather than waiting passively for someone to notice. The Sold-Out Signal Bus broadcasts availability deltas to the catalog and search layer so a listing page reflects “sold out” or “back in stock” within seconds of the counter store’s state actually changing.

A reservation’s life looks like this: a shopper starts checkout, the gateway assigns an idempotency key and passes the request to the routing engine, which picks one or more warehouses that together can cover the requested quantity. The gateway issues an atomic reserve against the counter store for each chosen warehouse, and only if every warehouse’s reserve succeeds does the gateway write a held row to the ledger with an expiry timestamp; if any warehouse’s reserve fails partway through a split allocation, the gateway rolls back the warehouses that already succeeded. From there, one of two things happens: the payment webhook confirms success and the gateway commits the reservation, permanently decrementing on-hand stock and marking the ledger row committed, or the TTL lapses without a commit and the sweeper releases the held quantity back to each warehouse’s available count and marks the ledger row expired. Every state change publishes an availability delta onto the signal bus, and once a SKU’s summed available count across all fifty warehouses hits zero, the bus fires a sold-out signal to the catalog.

Component Deep Dives

The Reservation Gateway and Idempotency Layer

This component’s job is to make sure the exact same client request, submitted twice because of a slow network or a retried mobile app, never creates two holds or triggers two releases.

The obvious mistake is trusting the client to only ever send a request once. Mobile networks drop connections mid-request constantly, and a client that times out waiting for a response has no way to know whether the server actually processed the request before the connection died, so a correct client has to retry, and a correct server has to treat that retry as harmless. We require every reserve, commit, and release call to carry a client-generated idempotency_key, and the gateway checks that key against a short-lived store before doing any work.

# Idempotency check performed before any reservation work begins
# Demonstrates: deduplicating a retried request using a Redis SETNX
# with the original response cached so a retry replays it verbatim
import json
import redis

IDEMPOTENCY_TTL_SECONDS = 86400  # keep replay protection for 24 hours

def with_idempotency(redis_client: redis.Redis, idempotency_key: str, handler):
    cache_key = f"idem:{idempotency_key}"
    lock_key = f"idem-lock:{idempotency_key}"

    cached = redis_client.get(cache_key)
    if cached is not None:
        return json.loads(cached)

    acquired = redis_client.set(lock_key, "1", nx=True, ex=30)
    if not acquired:
        # Another instance of this exact request is already in flight.
        raise DuplicateInFlightError(idempotency_key)

    try:
        result = handler()
        redis_client.set(cache_key, json.dumps(result), ex=IDEMPOTENCY_TTL_SECONDS)
        return result
    finally:
        redis_client.delete(lock_key)

Think of this like a coat check ticket at a busy restaurant: showing the same stub twice gets you the same coat handed back, never a second coat pulled off the rack. What breaks without it: a shopper on a spotty connection taps “reserve” once, the request goes through, the response is lost, the client retries, and now two holds exist for one checkout session, quietly eating into a SKU’s available count for stock that was never actually double-requested by a human.

The Distributed Inventory Counter Store

This component’s job is to hold the authoritative available, reserved, and on-hand counts for every warehouse/SKU pair and guarantee that a decrement can never push available below zero, no matter how many requests arrive for the same unit at the same instant.

The non-obvious part is that “atomic decrement” is not one technique, it is a choice between two families, and the right choice depends on how contended a given SKU currently is. For the overwhelming majority of SKUs, contention on any single warehouse/SKU pair is low enough that an optimistic approach works well: read the current available count and a version number, attempt an update guarded by that version, and retry from a fresh read if another writer won the race in between.

-- Optimistic reservation: version-stamped compare-and-swap
UPDATE inventory_counters
SET available = available - $1,
    reserved = reserved + $1,
    version = version + 1
WHERE warehouse_id = $2
  AND sku = $3
  AND version = $4
  AND available >= $1;

# Optimistic-locking retry loop for a warehouse/SKU pair under normal contention
# Demonstrates: read-modify-write guarded by a version column, retrying
# against fresh state whenever a concurrent writer wins the race
def reserve_optimistic(conn, warehouse_id: str, sku: str, qty: int, max_retries: int = 5) -> bool:
    for _ in range(max_retries):
        row = conn.execute(
            "SELECT available, version FROM inventory_counters WHERE warehouse_id = %s AND sku = %s",
            (warehouse_id, sku),
        ).fetchone()
        if row is None or row.available < qty:
            return False

        updated = conn.execute(
            """
            UPDATE inventory_counters
            SET available = available - %s, reserved = reserved + %s, version = version + 1
            WHERE warehouse_id = %s AND sku = %s AND version = %s
            """,
            (qty, qty, warehouse_id, sku, row.version),
        )
        if updated.rowcount == 1:
            conn.commit()
            return True
        # Another writer won the race for this row; loop and retry against fresh state.
    return False

A small number of SKUs, the doorbuster items and flash-sale exclusives, see the opposite pattern: thousands of concurrent requests target the same handful of warehouse/SKU rows in the same second, and an optimistic retry loop under that load spends most of its time retrying instead of making progress, since almost every attempt collides with another one. For those hot rows we switch to a pessimistic strategy backed by a single atomic operation, executed as a Redis Lua script so the check and the decrement happen as one indivisible step with no window for a second request to slip in between.

-- Atomic check-and-reserve against a per-warehouse, per-SKU counter
-- KEYS[1] = inventory key, e.g. "inv:{warehouse_id}:{sku}"
-- ARGV[1] = quantity requested
-- ARGV[2] = reservation TTL in seconds
-- ARGV[3] = reservation_id, used to key the hold record
local available = tonumber(redis.call('HGET', KEYS[1], 'available') or '0')
local qty = tonumber(ARGV[1])

if available < qty then
  return {0, available}
end

redis.call('HINCRBY', KEYS[1], 'available', -qty)
redis.call('HINCRBY', KEYS[1], 'reserved', qty)

local hold_key = 'hold:' .. ARGV[3]
redis.call('HSET', hold_key, 'warehouse_key', KEYS[1], 'qty', qty, 'status', 'held')
redis.call('EXPIRE', hold_key, ARGV[2])

local expiry_score = tonumber(redis.call('TIME')[1]) + tonumber(ARGV[2])
redis.call('ZADD', 'reservation_expiry', expiry_score, ARGV[3])

return {1, available - qty}

This is the same tradeoff a nightclub makes between a bouncer counting heads one at a time at the door versus a fire marshal’s posted maximum occupancy sign that everyone is trusted to self-police: the bouncer (pessimistic, one gate, no double-counting possible) is slower per person but never lets the room over capacity, while the honor-system sign (optimistic, fast, but occasionally two people squeeze through at once and someone has to be turned back) scales better when the room is nowhere near full. What breaks if every SKU used the pessimistic path: a single Redis key becomes a serialization point for every request against it, and for the 99% of SKUs that never see meaningful contention, that serialization buys nothing but adds latency for no reason.

The Fulfillment Routing Engine

This component’s job is to decide which of the fifty warehouses, or which combination of them, actually supplies a reservation’s requested quantity, so a shopper is never told “sold out” locally when the network as a whole has stock sitting a few states away.

A naive design reserves only against the single nearest warehouse and returns “unavailable” the moment that one warehouse comes up short, even if forty-nine other warehouses are fully stocked. That throws away real, sellable inventory because of where it happens to physically sit. We instead score every candidate warehouse against distance, outbound shipping cost, and available quantity, then greedily allocate the requested quantity across the top-ranked candidates until the full quantity is covered, splitting into more than one warehouse only when no single warehouse can cover the whole order alone.

# Cross-warehouse fulfillment routing: score candidate warehouses by
# distance, cost, and available stock, splitting an order across
# warehouses only when no single warehouse can cover it alone
from dataclasses import dataclass

@dataclass
class WarehouseCandidate:
    warehouse_id: str
    available_qty: int
    distance_km: float
    outbound_cost_per_unit: float

def score_candidate(candidate: WarehouseCandidate, requested_qty: int, max_distance_km: float) -> float:
    distance_term = 1.0 - min(candidate.distance_km / max_distance_km, 1.0)
    cost_term = 1.0 - min(candidate.outbound_cost_per_unit / 50.0, 1.0)
    fill_term = min(candidate.available_qty / requested_qty, 1.0)
    return 0.45 * distance_term + 0.25 * cost_term + 0.30 * fill_term

def route_reservation(
    candidates: list[WarehouseCandidate],
    requested_qty: int,
    max_distance_km: float = 800.0,
) -> list[tuple[str, int]]:
    ranked = sorted(
        candidates,
        key=lambda c: score_candidate(c, requested_qty, max_distance_km),
        reverse=True,
    )

    allocations: list[tuple[str, int]] = []
    remaining = requested_qty
    for candidate in ranked:
        if remaining <= 0:
            break
        if candidate.available_qty <= 0:
            continue
        take = min(candidate.available_qty, remaining)
        allocations.append((candidate.warehouse_id, take))
        remaining -= take

    if remaining > 0:
        return []  # network-wide stock is insufficient; caller signals sold out

    return allocations

Think of this like an air traffic controller assigning arriving flights to open runways: the closest runway is preferred, but if it cannot take the whole queue, the controller splits arrivals across a second runway rather than telling half the flights to circle indefinitely. What breaks without split allocation: a customer ordering a quantity of six loses the entire sale the moment the nearest warehouse has only four in stock, even though a second warehouse eighty miles away easily has the other two.

The Reservation Ledger and State Machine

This component’s job is to be the durable, queryable record of every reservation’s lifecycle, separate from the fast-but-ephemeral counter store, so a support agent or a reconciliation job can answer “what happened to this hold” long after the counter store has moved on.

The reservation-versus-commit distinction is the core of the whole design: a reservation is not a sale, it is a promise with an expiration date. A naive system that decrements on-hand stock the moment a cart is created loses stock to abandoned carts indefinitely; a system that only decrements at payment success leaves a window, between “shopper clicked buy” and “payment gateway confirmed,” where the same unit can be shown as available to someone else. The two-phase flow, hold first and commit or release second, closes that window while still letting abandoned carts give their stock back automatically.

-- Reservation ledger: the durable, queryable record of every hold's lifecycle
CREATE TABLE reservations (
    reservation_id      UUID        PRIMARY KEY,
    sku                 TEXT        NOT NULL,
    requested_qty       INT         NOT NULL CHECK (requested_qty > 0),
    status              TEXT        NOT NULL DEFAULT 'held'
        CHECK (status IN ('held', 'committed', 'released', 'expired')),
    idempotency_key     TEXT        NOT NULL UNIQUE,
    created_at          TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    expires_at          TIMESTAMPTZ NOT NULL,
    committed_at        TIMESTAMPTZ,
    released_at         TIMESTAMPTZ,
    payment_id          TEXT
);

CREATE INDEX ON reservations (status, expires_at) WHERE status = 'held';
CREATE INDEX ON reservations (payment_id) WHERE payment_id IS NOT NULL;

A hold’s only legal transitions are held -> committed, held -> released, and held -> expired; once a reservation leaves held it is terminal, and the ledger write function enforces that with a guarded update rather than trusting callers to check state first.

# Reservation state transition, enforced as a single guarded update
# Demonstrates: illegal transitions (e.g. committing an already-expired
# hold) are rejected at the database level, not just in application code
def commit_reservation(conn, reservation_id: str, payment_id: str) -> bool:
    updated = conn.execute(
        """
        UPDATE reservations
        SET status = 'committed', committed_at = NOW(), payment_id = %s
        WHERE reservation_id = %s AND status = 'held'
        """,
        (payment_id, reservation_id),
    )
    conn.commit()
    return updated.rowcount == 1

The TTL Expiry Sweeper

This component’s job is to actively find reservations whose TTL has lapsed and release their held stock back into the available pool, rather than waiting for a client to ask about that stock again before noticing it should have been freed.

Relying only on lazy expiry, checking whether a hold has expired the next time someone happens to read it, sounds simpler, but it means a SKU’s available count stays artificially low until some unrelated read stumbles across the stale hold. During a flash sale, where the sold-out signal needs to reflect true availability within seconds, that lag is unacceptable: a SKU could show as sold out for minutes after its last hold quietly expired, costing real sales. The sweeper runs on its own schedule, actively scanning for expired holds instead of waiting to be asked.

// TTL expiry sweeper: claims expired reservations from a Redis sorted
// set and releases their held stock back to the available pool
package sweeper

import (
	"context"
	"strconv"
	"time"

	"github.com/redis/go-redis/v9"
)

const (
	batchSize   = 200
	claimTTL    = 30 * time.Second
	sweepPeriod = 2 * time.Second
)

func (s *Sweeper) Run(ctx context.Context, rdb *redis.Client) error {
	ticker := time.NewTicker(sweepPeriod)
	defer ticker.Stop()

	for {
		select {
		case <-ctx.Done():
			return ctx.Err()
		case <-ticker.C:
			if err := s.sweepOnce(ctx, rdb); err != nil {
				s.logger.Error("sweep failed", "err", err)
			}
		}
	}
}

func (s *Sweeper) sweepOnce(ctx context.Context, rdb *redis.Client) error {
	now := float64(time.Now().Unix())
	expired, err := rdb.ZRangeByScore(ctx, "reservation_expiry", &redis.ZRangeBy{
		Min:   "0",
		Max:   strconv.FormatFloat(now, 'f', 0, 64),
		Count: batchSize,
	}).Result()
	if err != nil || len(expired) == 0 {
		return err
	}

	for _, reservationID := range expired {
		claimed, err := rdb.SetNX(ctx, "claim:"+reservationID, s.workerID, claimTTL).Result()
		if err != nil || !claimed {
			continue // another sweeper worker already owns this reservation
		}
		if err := s.releaseReservation(ctx, rdb, reservationID); err != nil {
			s.logger.Error("release failed", "reservation_id", reservationID, "err", err)
			continue
		}
		rdb.ZRem(ctx, "reservation_expiry", reservationID)
	}
	return nil
}

The analogy is a parking enforcement officer who actively walks the lot checking meters, rather than a system that only notices an expired meter the next time a different driver happens to try that same spot. Running several sweeper workers in parallel is necessary at this scale, since a single worker scanning 200 expired holds every two seconds cannot keep up during a flash sale’s expiry wave, but multiple workers racing for the same expired reservation would double-release the same stock without the SetNX-based claim shown above acting as a fencing token, guaranteeing exactly one worker processes any given expired hold.

The Sold-Out Signal Bus

This component’s job is to tell the catalog and search layer, within seconds, when a SKU’s network-wide available stock reaches zero, and when it becomes available again.

The obvious approach, having the catalog poll the counter store directly on every page load, does not scale: a popular SKU’s product page can receive tens of thousands of views a minute, and hammering the counter store with read traffic on top of its write load competes with the very reservations it needs to process quickly. Instead, every state change in the counter store publishes a small availability-delta event onto a stream, and the catalog layer maintains its own cheap, eventually-consistent view built from that stream rather than querying the counter store live.

A SKU’s summed available count fluctuating around zero during a flash sale, as holds are created and released within milliseconds of each other, would cause a naive sold-out signal to flap on and off dozens of times a minute if it fired on every single delta. We debounce it: a sold-out signal only fires once available stays at zero across a short hysteresis window, and a back-in-stock signal only fires once available climbs back above a small buffer, not the instant it ticks up by one.

Data Model

The data model spans four entities: the inventory counters themselves, the reservation ledger, per-warehouse allocation records for split shipments, and an append-only event log for auditing.

-- Inventory counters: the authoritative available/reserved/on-hand state
-- per warehouse per SKU, the row every reserve/commit/release touches
CREATE TABLE inventory_counters (
    warehouse_id    TEXT        NOT NULL,
    sku             TEXT        NOT NULL,
    available       INT         NOT NULL CHECK (available >= 0),
    reserved        INT         NOT NULL CHECK (reserved >= 0),
    on_hand         INT         NOT NULL CHECK (on_hand >= 0),
    version         BIGINT      NOT NULL DEFAULT 0,
    updated_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    PRIMARY KEY (warehouse_id, sku)
);

CREATE INDEX ON inventory_counters (sku) INCLUDE (available);

-- Per-warehouse allocation for a single reservation, one row per
-- warehouse the fulfillment router decided to split the order across
CREATE TABLE reservation_allocations (
    id              BIGSERIAL   PRIMARY KEY,
    reservation_id  UUID        NOT NULL REFERENCES reservations(reservation_id),
    warehouse_id    TEXT        NOT NULL,
    allocated_qty   INT         NOT NULL CHECK (allocated_qty > 0),
    UNIQUE (reservation_id, warehouse_id)
);

CREATE INDEX ON reservation_allocations (warehouse_id);

-- Append-only audit log backing reconciliation and support investigation
CREATE TABLE reservation_events (
    id              BIGSERIAL   PRIMARY KEY,
    reservation_id  UUID        NOT NULL,
    event_type      TEXT        NOT NULL
        CHECK (event_type IN ('held', 'committed', 'released', 'expired')),
    warehouse_id    TEXT,
    qty             INT,
    occurred_at     TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE INDEX ON reservation_events (reservation_id, occurred_at);

inventory_counters is sharded by hashing warehouse_id and sku together, not by sku alone. Sharding by SKU alone would put all fifty of a hot doorbuster SKU’s warehouse rows near each other in the hash space, since they share the same SKU string, concentrating every warehouse’s traffic for that one SKU onto a small handful of shards right when contention is highest. Hashing the combined key spreads the same SKU’s fifty warehouse rows across the whole cluster instead, so a single popular item never creates a single hot shard.

A reservation is born the moment checkout starts: the routing engine produces one or more reservation_allocations rows, the counter store atomically decrements available and increments reserved for each allocated warehouse, and the ledger writes a held row with its expires_at timestamp. From there exactly one of two paths fires. On the happy path, the payment webhook confirms success, the ledger flips to committed, and each allocated warehouse’s reserved count decrements while on_hand decrements permanently. On the release path, either the TTL sweeper finds an expired hold or the shopper explicitly cancels checkout, the ledger flips to expired or released, and each allocated warehouse’s reserved count decrements while available increments back to where it started.

Key Algorithms and Protocols

Atomic Check-and-Reserve via Redis Lua Script

Covered in the Distributed Inventory Counter Store section above, this algorithm runs the availability check and the decrement as a single indivisible operation on the Redis event loop, so no second request can read a stale available value in the gap between the check and the write.

Cross-Warehouse Greedy Allocation

Covered in the Fulfillment Routing Engine section above, this algorithm ranks candidate warehouses by a weighted score of distance, cost, and fill potential, then greedily allocates the requested quantity across the top-ranked candidates in O(n log n) time for n candidate warehouses, dominated by the initial sort.

TTL Sweep Claim-and-Process

Covered in the TTL Expiry Sweeper section above, this algorithm scans a Redis sorted set ordered by expiry timestamp in fixed-size batches, uses a SetNX-based fencing claim so multiple sweeper workers never process the same expired reservation twice, and releases held stock back to the counter store for each claimed reservation.

Optimistic CAS versus Pessimistic Lock

Covered in the Distributed Inventory Counter Store section above, the choice between a version-guarded compare-and-swap update and a single atomic Lua-scripted decrement is made per SKU based on measured contention, not fixed at design time for the whole catalog.

Scaling and Performance

The Distributed Inventory Counter Store is the component under the most sustained write load, since every reservation, commit, and release touches it, and it scales by combining two techniques: sharding the counter cluster by a hash of warehouse_id and sku together, and splitting any individual SKU that becomes a hot spot into several virtual sub-counters.

A hot SKU during a flash sale can see so much concurrent traffic against one warehouse’s single counter row that even an atomic Lua script becomes a serialization bottleneck, since every request against that key still has to wait its turn on Redis’s single-threaded command execution for that key. We split such a SKU’s counter into a fixed number of virtual sub-counters (inv:{warehouse_id}:{sku}:0 through :9), have each incoming request pick a sub-counter at random for its decrement attempt, retry a different sub-counter if the one it picked is out of stock, and sum all ten sub-counters together whenever a read needs the true total.

Capacity Estimation:

Given:
  - 50 warehouses, ~2,000,000 SKUs network-wide
  - Peak reservation rate: 80,000 reservations/sec (flash sale)
  - Default TTL: 15 minutes (900 seconds)
  - Average payload per hold record: ~200 bytes

Concurrent open holds at peak (Little's Law: L = lambda * W):
  80,000 reservations/sec * 900 sec = 72,000,000 concurrent open holds

Hold record memory footprint:
  72,000,000 holds * 200 bytes = ~14.4 GB
  Comfortably shardable across a Redis cluster of a few dozen nodes

Counter row footprint:
  50 warehouses * 2,000,000 SKUs = 100,000,000 counter rows
  100,000,000 rows * ~64 bytes/row (hash with 3 numeric fields) = ~6.4 GB

Ledger write throughput:
  80,000 reserves/sec + ~80,000 commits-or-releases/sec (each hold
  resolves exactly once) = ~160,000 durable ledger writes/sec at peak,
  partitioned by reservation_id hash across the ledger's write nodes

Event bus throughput:
  Each reservation emits ~3 events across its lifecycle (held, then
  committed or released, plus one availability delta) =
  80,000 * 3 = 240,000 events/sec at peak, each a small (<500 byte)
  message - well within range for a commodity pub/sub system.

The dominant bottleneck is not aggregate throughput, the numbers above are modest by distributed-systems standards, it is the small number of individual hot rows that see disproportionate concurrent traffic during a flash sale. Sharding by warehouse-plus-SKU hash spreads ordinary load evenly, and virtual sub-counter splitting handles the small number of rows where even a perfectly atomic single-key operation still becomes a serialization point.

Failure Modes and Recovery

Failure	Detection	Impact	Recovery
Redis node holding a hot SKU’s counter goes down mid-decrement	Connection timeout on the counter shard, replica failover alarm	Reserve requests against that SKU fail or stall until failover completes	Lua scripts are atomic and AOF-persisted before acknowledgment; a replica promotes and resumes serving the same key range, and the idempotency layer makes client retries safe
Sweeper falls behind or crashes, leaving expired holds unreleased	Growing lag on the reservation_expiry sorted set, alerting on backlog size	Available stock is understated network-wide; SKUs appear sold out when stock is actually just held by expired reservations	Multiple sweeper workers run concurrently with SetNX claim fencing, so remaining workers absorb a crashed worker’s share of the backlog without double-releasing
Payment webhook lost or delivered twice	Idempotency key mismatch on commit, or a held reservation past its TTL with no matching payment confirmation	A paid order silently expires and releases stock the customer already paid for, or a duplicate webhook double-commits	Commit is idempotent, keyed by payment_id; a reconciliation job cross-checks the payment gateway’s ledger against reservations.payment_id on a schedule to catch mismatches
Network partition separates a counter shard from the ledger	Ledger write timeout immediately following a successful counter decrement	A hold exists in the counter store’s memory but has no durable record, at risk of being lost if that counter node restarts	Gateway requires the ledger write to succeed before acknowledging the reservation to the client; a counter decrement without a matching ledger row is treated as invalid and rolled back by a background consistency check
Clock skew between regions affects TTL expiry timing	Reservations expiring noticeably earlier or later than their configured TTL relative to wall-clock creation time	Some holds release prematurely (false sold-out relief) or linger too long (falsely blocked stock)	TTL expiry is computed once, at reservation time, as an absolute epoch timestamp stored in the sorted set, never as a relative duration recomputed later against a different node’s clock
Hot SKU causes single-shard contention during a flash sale	p99 latency spike and CPU saturation isolated to one shard	Reserve requests for that one SKU queue or time out while the rest of the catalog is unaffected	Hot SKU counter splitting into virtual sub-counters, triggered automatically once a SKU’s request rate crosses a threshold, spreads its load across multiple shards

Comparison of Approaches

Approach	Latency	Complexity	Failure mode	Best fit
Decrement only on payment success, no reservation	Fastest at checkout start (no hold created)	Low	Overselling during the gap between cart and payment confirmation	Low-traffic catalogs where the oversell risk is negligible
Hard lock per SKU held until explicit release	Fast once acquired, but blocks other requests while held	Medium	A crashed or abandoned client holds stock hostage indefinitely	Internal tooling with trusted, well-behaved clients, rarely applicable to public checkout
Optimistic CAS retry loop for every SKU, regardless of contention	Fast under low contention, degrades sharply under high contention	Medium	Hot SKUs spend most cycles retrying instead of making progress	Catalogs with uniformly low per-SKU contention and no flash-sale traffic pattern
Soft TTL reservation with per-SKU optimistic or pessimistic strategy (this design)	Sub-50ms reserve, bounded worst case even under hot-SKU load	Medium-high	Requires careful TTL and hysteresis tuning, plus reconciliation tooling	Multi-warehouse retail at scale, with both long-tail and flash-sale SKUs in the same catalog
Two-phase commit across all warehouses for every reservation	Slowest, blocked on the slowest participant warehouse	Very high	A single slow or unreachable warehouse stalls the entire reservation, even for the warehouses that responded instantly	Rarely justified at this scale; useful only where strict cross-warehouse atomicity outweighs latency

We would pick the soft TTL reservation design with a per-SKU concurrency strategy for any retailer operating more than a handful of warehouses with a real mix of everyday and flash-sale demand, because it is the only approach here that keeps the correctness guarantee, available never goes negative, while still hitting a sub-50ms latency target under both normal and extreme contention. The alternatives each fail on a different axis: no reservation at all oversells, a hard lock risks stranding stock forever, a single fixed concurrency strategy either wastes latency on uncontended SKUs or collapses under contended ones, and full two-phase commit trades away latency for a consistency guarantee stronger than this problem actually needs.

Key Takeaways

• A reservation is a soft, self-expiring hold, not a hard lock. A TTL means a crashed or abandoned client can never strand stock indefinitely; the system heals itself without anyone noticing.
• Atomic decrement is not one technique, it is a per-SKU choice. Optimistic version-CAS suits the long tail of low-contention SKUs; a pessimistic atomic Lua script suits the small number of hot, contended ones.
• The reservation-versus-commit split closes the exact window a naive single-step decrement leaves open. Stock is held the instant checkout begins and only permanently decremented once payment actually confirms.
• Cross-warehouse fulfillment routing has to run before any warehouse’s counter is touched, scoring and allocating across candidates up front rather than trying warehouses one at a time and retrying on failure.
• TTL expiry needs an active sweeper, not just lazy expiry-on-read. A sold-out signal that depends on someone else’s unrelated read to notice an expired hold lags exactly when freshness matters most.
• Sold-out signaling must be hysteresis-gated, not a raw mirror of the counter store. A signal that fires on every fluctuation around zero flickers on and off during exactly the traffic pattern, a flash sale, where shoppers most need a stable answer.
• Sharding the counter store by warehouse-plus-SKU, not SKU alone, prevents a popular item from concentrating its own load onto a handful of shards.
• A hot SKU’s counter can outgrow even an atomic single-key operation. Splitting it into virtual sub-counters, summed on read, is the same technique high-throughput ticketing systems use for the same reason.

The counter-intuitive lesson is that the hardest-sounding requirement, never overselling under massive concurrency, is actually solved by a fairly small and well-understood atomic operation. The harder engineering problem is everything around that operation: choosing the right concurrency strategy per SKU instead of one strategy for the whole catalog, making the TTL self-healing instead of relying on a client to behave, and making the sold-out signal reflect reality without flickering on every millisecond-scale fluctuation in a busy system.

Frequently Asked Questions

Q: Why not just decrement inventory on payment success instead of reserving upfront?

A: Because payment confirmation can take anywhere from a couple of seconds to a couple of minutes, and during that window a naive system shows the full pre-checkout stock as available to every other shopper. Two shoppers can both be told an item is in stock, both complete payment, and only one of them can actually receive the unit. Reserving upfront closes that window by removing the requested quantity from the available pool the moment checkout begins, not the moment payment clears.

Q: Why not use a single global distributed lock, like a ZooKeeper or etcd lock, per SKU instead of atomic Redis counters?

A: A distributed lock forces every request against a SKU to acquire and release the lock serially, which caps throughput at whatever one lock holder can process at a time, and it introduces an entirely new failure mode: a client that acquires the lock and then crashes or stalls holds every other request hostage until the lock’s own timeout fires. An atomic counter operation, whether the Lua-scripted version or the version-CAS version, achieves the same correctness guarantee without ever requiring a client to hold anything open across a network round trip.

Q: How do you prevent double-release when multiple sweeper instances run concurrently?

A: Each sweeper worker attempts a SetNX-based claim on a reservation before releasing it, and only the worker that successfully sets that claim key proceeds. Every other worker sees the claim already exists and skips that reservation. The claim itself has a short TTL, so a worker that crashes mid-release does not permanently orphan the reservation, it simply becomes claimable again once the claim expires.

Q: What happens when a customer’s cart spans SKUs that live in different warehouses, or even the same SKU split across warehouses?

A: Each SKU line in the cart runs its own routing decision independently, and a single SKU that needs splitting produces multiple rows in reservation_allocations under one reservation_id. The reservation as a whole is only considered held once every warehouse’s allocation succeeds; if any one warehouse’s atomic reserve fails partway through, the gateway rolls back the allocations that already succeeded rather than leaving a partially-held reservation in the ledger.

Q: Why 15 minutes for the default TTL, and what tradeoffs does that duration choice raise?

A: Fifteen minutes comfortably covers the vast majority of legitimate checkout sessions, including a shopper who needs to dig out a card or fix a typo, without holding stock hostage for an unreasonably long time if they simply abandon the cart. A shorter TTL frees up contested stock faster during a flash sale but risks expiring a hold out from under a slow but genuinely still-checking-out shopper; a longer TTL is friendlier to slow shoppers but lets abandoned carts suppress availability for longer. The TTL is configurable per campaign specifically so a flash sale can run a shorter window than everyday browsing.

Q: How do you handle a hot SKU during a flash sale without a single Redis node becoming a bottleneck?

A: Once a SKU’s request rate crosses a configured threshold, its counter is automatically split into a fixed number of virtual sub-counters distributed across different shards, and incoming requests pick a sub-counter at random, retrying a different one if the one they picked happens to be depleted. Reads sum all sub-counters together to report the true total. This spreads what would otherwise be one overloaded key’s traffic across several physically separate keys.

Interview Questions

Q: Walk through what happens, end to end, when a shopper clicks “reserve” on the last unit of a popular SKU that two other shoppers are also trying to buy at the same instant.

Expected depth: Cover the idempotency check at the gateway, the fulfillment router’s warehouse selection, and why the actual conflict resolution happens inside the atomic Lua script or version-CAS update against the counter store, not anywhere earlier in the request path. Discuss why exactly one of the three concurrent requests succeeds and the other two receive a clean “unavailable” response rather than a partial or corrupted count.

Q: How would you decide, as a design choice rather than a fixed rule, whether a given SKU should use the optimistic or the pessimistic reservation strategy?

Expected depth: Discuss measuring per-SKU request contention in real time, the tradeoff between retry overhead (optimistic, cheap under low contention, wasteful under high contention) and serialization overhead (pessimistic, safe under any contention level but adds latency when contention was never actually a problem), and what signal would trigger an automatic switch from one strategy to the other for a SKU that suddenly goes viral.

Q: A reservation’s TTL expires at the exact moment its payment webhook arrives confirming success. Design the logic that decides the outcome.

Expected depth: Cover the guarded state transition (UPDATE ... WHERE status = 'held') that ensures only one of “commit” or “expire” can win, discuss why the losing operation should fail cleanly rather than silently overwrite the winner, and address what the customer-facing outcome should be if payment wins a race against an already-expired hold versus if the hold expires first and payment arrives moments later needing a manual reconciliation path.

Q: The system needs to support a flash sale where a single SKU will see 50,000 reservation attempts per second for fifteen minutes straight. What changes, if anything, from the steady-state design?

Expected depth: Discuss triggering virtual sub-counter splitting ahead of the sale rather than reactively, tuning the sold-out signal’s hysteresis window tighter so shoppers get an accurate answer faster, and whether the TTL itself should shrink for that specific campaign to recycle contested stock faster. Touch on why the reservation ledger’s write throughput, not the counter store, might become the actual bottleneck once a single SKU dominates traffic this heavily.

Q: How would you extend this design if the business wanted to let a shopper’s cart hold reservations across two separate checkout attempts, for example if they abandon checkout but return within the same 15-minute window?

Expected depth: Discuss whether the existing reservation should simply be reused (matched by cart or session identifier) rather than releasing and re-reserving, the risk of extending a TTL indefinitely every time a shopper touches the cart, and why a bounded maximum hold duration independent of activity is necessary to prevent a determined shopper from holding scarce stock far longer than the TTL was ever meant to allow.