A thousand-page reference manual never makes you start on page one to find page four hundred. It has a table of contents that points you at the nearest chapter start, and once you land there you skim forward a few paragraphs to the exact sentence you wanted. Nobody rereads the whole book to jump to the middle of it.

Scrubbing a game replay to the fourteen-minute mark needs exactly the same trick. A finished match is really a long sequence of tiny state changes: a player moves, a weapon fires, a shield breaks, a round ends. Play all of those back in order from the beginning and you get a perfect reconstruction of the match, but if a viewer wants to jump straight to the game-winning play at minute fourteen, replaying thirteen minutes and fifty-nine seconds of prior events just to get there is not an option. We need something like that table of contents: a way to land near the right moment instantly, then step forward only a handful of small updates to reach the exact instant the viewer asked for.

The system has to do this at a scale that makes the naive answers fall apart fast. Every match on the platform generates a continuous stream of gameplay events that has to be captured durably, without adding latency to the live simulation loop that is also, at that exact moment, rendering the next frame for every player in the match. Once a match ends, its replay has to become scrubbable to any point in under a second, and it has to stay scrubbable for millions of playback sessions a day, spread across recent matches, viral highlight clips, and tournament VODs, without every one of those sessions hitting the same origin servers that also run live matches.

Three forces are in tension here. Capture has to be effectively free: recording an event can never slow down the tick that produced it. Storage has to be efficient: recording the full game state on every single tick would make storage costs balloon, but the format still has to support near-instant seeking, which pulls in the opposite direction. And delivery has to scale independently of origin capacity: millions of daily replay views cannot each be a fresh round trip to the servers that wrote the file. We need to solve for non-blocking event capture on a live game server, a file format that mixes full state snapshots with cheap incremental updates so seeking never means replaying from the start, and a delivery path that lets a CDN absorb almost all playback traffic, all three at once.

Requirements and Constraints

Functional Requirements

• Capture every gameplay event (movement, actions, damage, state transitions) generated by a live match into a durable, ordered log
• Assemble a finished match’s event log into a single replay file once the match ends
• Let a viewer scrub to any timestamp in a completed replay and begin playback from that exact point
• Support common playback controls built on scrubbing: play, pause, fast-forward, rewind, and jump-to-event (for example, jump to the next kill or the start of the next round)
• Serve replay playback to spectators, VOD viewers, and broadcast tooling without those requests reaching the live game servers
• Compress recorded event and snapshot data for storage efficiency while preserving fast, granular seeking
• Move replays through a storage lifecycle: fresh and fast to access right after the match, cheaper and slower after that, archived indefinitely only when flagged, deleted otherwise

Non-Functional Requirements

• Recording overhead: event capture on the game server adds no more than 0.5ms to any single simulation tick, and never blocks the tick loop waiting on I/O
• Scale: roughly 20 million matches/day, average match length 20 minutes (1,200 seconds), average 40 players/match, an aggregate delta-event rate of about 50 events/sec per match
• Seek latency: p99 under 1 second for a cold (uncached) seek anywhere in a finished replay; under 150ms for a seek served from a warm CDN cache
• Finalization latency: a replay becomes playable within 30 seconds of the match ending
• Playback scale: roughly 8 million replay playback sessions/day, each issuing on average 15 seek or range requests, for about 120 million total range requests/day
• Origin protection: at least 97% of replay playback bytes served from CDN edge cache, never touching the origin replay servers
• Durability: once a replay is finalized, 99.999999999% durability (S3-class) until its scheduled deletion or archival
• Availability: 99.9% availability for replay playback; the recording path fails toward “record less, never block the game” rather than dropping frames of live gameplay

Constraints and Assumptions

• Out of scope: live in-progress spectating (watching a match as it happens). This system builds post-match replay only; live spectating is a related but separate real-time streaming problem
• We assume the game client can deterministically (or near-deterministically, with periodic full-state correction) reconstruct what a match looked like from its recorded events; guaranteeing that reconstruction is a game-engine responsibility, not something this storage system provides
• Voice chat and rendered video capture are out of scope. This is event-based replay data, not a video recording
• Anti-cheat review and moderation are downstream consumers of the raw event log; we build the storage and retrieval layer they read from, not the review tooling itself
• We assume a single authoritative region per match (the region the game server ran in); cross-region replay access is served through the CDN, not a cross-region write path

High-Level Architecture

The system has five major components: the Event Recorder running inside the live game server process, the Replay File Writer that encodes captured events into a keyframe-and-delta file format, the Seek Index Builder that produces a timestamp-to-byte-offset manifest for each finished replay, the Replay Server that handles session start and metadata, and the CDN Delivery Layer that serves the actual replay bytes to viewers.

During a live match, the Event Recorder sits inside the game server’s process and buffers every event (a move, a cast, a damage tick) into an in-memory ring buffer, without ever making the simulation tick wait on disk or network I/O. A separate flush goroutine drains that buffer on a fixed interval and ships batches to the Replay File Writer, which is what actually decides, per batch, whether to emit a full keyframe (a complete snapshot of match state) or a small delta block (the incremental changes since the last snapshot). When the match ends, the writer finalizes the file, and the Seek Index Builder walks the finished file to produce a compact manifest mapping playback timestamps to byte offsets. That manifest, plus the replay file itself, gets pushed to the CDN. From that point forward, a viewer’s player client fetches the manifest once from the Replay Server at session start, then issues HTTP range requests directly against the CDN edge to scrub anywhere in the match, without the Replay Server or the original game server being involved at all.

That last point is the crux of the whole design: the seek index is not a database the Replay Server queries on every scrub, it is a small file published alongside the replay itself, so that after the first request of a session, the entire playback experience runs against CDN-cached bytes.

Event Recording on the Live Game Server

This component’s job is to get every gameplay event out of the simulation loop and into a durable pipeline without the simulation loop ever noticing it happened.

The naive instinct is to write each event straight to a database or a network socket the moment it occurs, because that is the simplest code to write and it means nothing is ever lost. The problem is that the simulation tick that produces these events is also the tick computing physics, hit detection, and the next frame every player in the match is about to see. Any synchronous I/O call on that thread, even a fast one, adds jitter to gameplay, and jitter on a 60Hz or 128-tick server is something players feel immediately. The fix is to make event capture a pure, in-memory append with zero I/O, and move all I/O to a separate thread that the simulation loop never waits on.

// Game server: non-blocking event capture into an in-memory ring buffer
// Demonstrates: how the simulation tick loop hands off events without blocking on I/O
package recorder

import (
	"sync"
	"time"
)

type EventType uint8

const (
	EventMove EventType = iota
	EventDamage
	EventAbilityCast
	EventStateChange
)

type GameEvent struct {
	SeqNo    uint32 // monotonic, per-match, tick-based - never wall clock
	TickMs   uint32 // milliseconds since match start
	Type     EventType
	EntityID uint32
	Payload  []byte // small, event-type-specific encoded fields
}

// RingBuffer is a fixed-capacity, single-writer, single-reader buffer.
// The simulation goroutine only ever calls Push, which never allocates
// and never blocks - if the buffer is full, the oldest unflushed event
// is dropped and a counter is incremented instead of stalling the tick.
type RingBuffer struct {
	mu      sync.Mutex
	buf     []GameEvent
	head    int
	size    int
	dropped uint64
}

func NewRingBuffer(capacity int) *RingBuffer {
	return &RingBuffer{buf: make([]GameEvent, capacity)}
}

func (r *RingBuffer) Push(e GameEvent) {
	r.mu.Lock()
	defer r.mu.Unlock()
	idx := (r.head + r.size) % len(r.buf)
	if r.size == len(r.buf) {
		r.head = (r.head + 1) % len(r.buf) // overwrite oldest, count the drop
		r.dropped++
	} else {
		r.size++
	}
	r.buf[idx] = e
}

// DrainBatch is called by a separate flush goroutine on a fixed interval,
// never by the simulation thread itself.
func (r *RingBuffer) DrainBatch(max int) []GameEvent {
	r.mu.Lock()
	defer r.mu.Unlock()
	n := r.size
	if n > max {
		n = max
	}
	batch := make([]GameEvent, n)
	for i := 0; i < n; i++ {
		batch[i] = r.buf[(r.head+i)%len(r.buf)]
	}
	r.head = (r.head + n) % len(r.buf)
	r.size -= n
	return batch
}

// StartFlushLoop runs on its own goroutine, off the simulation tick,
// and ships batches to the ingest pipeline every flushInterval.
func StartFlushLoop(rb *RingBuffer, flushInterval time.Duration, ship func([]GameEvent) error) {
	ticker := time.NewTicker(flushInterval)
	go func() {
		for range ticker.C {
			batch := rb.DrainBatch(2000)
			if len(batch) == 0 {
				continue
			}
			if err := ship(batch); err != nil {
				// Ship failures never propagate back to the simulation loop;
				// the batch is re-queued for retry with backoff by the caller.
				continue
			}
		}
	}()
}

Think of the ring buffer like a stenographer taking shorthand notes during a live meeting: they write fast, cheap symbols in the moment and never stop to look anything up, leaving the actual transcription to happen later, off to the side, without disrupting the meeting. If the meeting talks faster than the stenographer can keep up, a real stenographer drops the least important filler, not the whole meeting; our ring buffer does the same by discarding the oldest unflushed events under sustained backpressure rather than blocking Push.

What would break if we simplified this to a single mutex-protected queue with unbounded growth instead of a fixed-size ring buffer? Under a burst (a team fight with forty players simultaneously casting abilities), the queue could grow without bound, and a slow flush consumer would let memory balloon on the game server process that is also running the live simulation, eventually causing the kind of GC pause or OOM that is far worse for gameplay than losing a few seconds of replay fidelity.

// Event batch shipped from the game server to the Replay File Writer
// Demonstrates: wire format for the async flush, tick-based timestamps
syntax = "proto3";

package replay.ingest.v1;

message EventBatch {
  int64 match_id = 1;
  uint32 first_seq_no = 2;
  repeated GameEventProto events = 3;
}

message GameEventProto {
  uint32 seq_no = 1;
  uint32 tick_ms = 2;    // milliseconds since match start, not wall clock
  uint32 event_type = 3;
  uint32 entity_id = 4;
  bytes payload = 5;
}

service ReplayIngest {
  rpc ShipEventBatch(EventBatch) returns (ShipAck);
}

message ShipAck {
  bool accepted = 1;
  uint64 server_dropped_count = 2; // events the ring buffer had to discard
}

The Replay File Writer: Keyframe and Delta Encoding

This component’s job is to turn a stream of received event batches into a single file on disk that can be scrubbed to any point without reading the whole thing.

A smart engineer’s first instinct is to just append every event to a file in order, no snapshots, nothing special. That works fine for sequential playback from the start, but it fails the actual requirement: scrubbing to minute fourteen of a twenty-minute match would mean reconstructing state by replaying almost the entire match’s worth of events first, which for a 128-tick server and forty players is not a sub-second operation. The fix is the same idea as the reference manual’s table of contents: periodically write a keyframe, a complete snapshot of every entity’s state at that instant, and write cheap delta events in between keyframes. Seeking then only ever means jumping to the nearest keyframe at or before the target and replaying the (small, bounded) number of deltas after it.

# Replay File Writer: mixes full keyframes with small delta events
# Demonstrates: keyframe interval selection and segment framing on disk
import struct

KEYFRAME_INTERVAL_MS = 15_000  # full snapshot every 15 seconds of match time

class ReplaySegmentWriter:
    """
    Writes a replay file as a sequence of segments:
      [segment_header][payload]
    segment_header = type(1 byte) + timestamp_ms(4 bytes) + payload_len(4 bytes)
    type 0x01 = keyframe (full entity state snapshot, compressed)
    type 0x02 = delta block (a short compressed run of small event deltas)
    """
    def __init__(self, file_handle):
        self.f = file_handle
        self.last_keyframe_ts = -KEYFRAME_INTERVAL_MS
        self.offset = 0
        self.index_entries = []  # (timestamp_ms, byte_offset, segment_type)

    def maybe_write_keyframe(self, timestamp_ms: int, full_state: bytes) -> bool:
        if timestamp_ms - self.last_keyframe_ts < KEYFRAME_INTERVAL_MS:
            return False
        self._write_segment(segment_type=0x01, timestamp_ms=timestamp_ms, payload=full_state)
        self.last_keyframe_ts = timestamp_ms
        return True

    def write_delta_block(self, timestamp_ms: int, delta_payload: bytes) -> None:
        self._write_segment(segment_type=0x02, timestamp_ms=timestamp_ms, payload=delta_payload)

    def _write_segment(self, segment_type: int, timestamp_ms: int, payload: bytes) -> None:
        header = struct.pack(">BII", segment_type, timestamp_ms, len(payload))
        self.f.write(header)
        self.f.write(payload)
        self.index_entries.append((timestamp_ms, self.offset, segment_type))
        self.offset += len(header) + len(payload)

The analogy that makes the tradeoff concrete is a road trip with a GPS that only logs your exact coordinates every fifteen minutes, plus a compass heading every second in between. To find out exactly where you were at minute fourteen, you do not need the full fifteen-minute GPS log, you need the nearest waypoint before minute fourteen and then you follow the compass headings forward from there. A shorter waypoint interval means less compass-following work per query, but many more full GPS fixes recorded overall; a longer interval means fewer full fixes but more compass headings to walk through per query.

What would break if we removed keyframes entirely and kept only deltas? Every seek would cost O(seek time) instead of O(keyframe interval), which is exactly the naive-append failure mode described above. What would break if we kept only keyframes and no deltas, snapshotting on every tick? Storage would balloon by orders of magnitude, since a full entity snapshot is dramatically larger than the handful of fields that actually changed since the last tick.

The Seek Index

This component’s job is to answer, for any requested playback timestamp, exactly which byte offset in the replay file to start reading from to satisfy that request in well under a second.

The instinct to avoid here is treating the seek index as an afterthought, something the Replay Server can compute on demand by scanning the file’s segment headers. Scanning headers sequentially to find the right one is still O(n) in the number of segments, and while that is fast for one request, it does not hold up under 120 million range requests a day. What we actually want is a compact, pre-built manifest that a binary search can walk in O(log n), published once per replay and small enough to ship down to the client alongside the replay itself.

# Seek Index: binary search from a requested playback timestamp to the
# nearest preceding keyframe offset, then the delta blocks needed after it
# Demonstrates: O(log n) lookup, bounded forward-decode cost
from bisect import bisect_right
from dataclasses import dataclass

@dataclass
class IndexEntry:
    timestamp_ms: int
    byte_offset: int
    segment_type: int  # 0x01 keyframe, 0x02 delta block

class SeekIndex:
    def __init__(self, entries: list[IndexEntry]):
        # Entries are stored sorted by timestamp_ms at write time
        self.entries = entries
        self.timestamps = [e.timestamp_ms for e in entries]

    def plan_seek(self, target_ms: int) -> tuple[IndexEntry, list[IndexEntry]]:
        # Find the latest segment at or before target_ms
        pos = bisect_right(self.timestamps, target_ms) - 1
        if pos < 0:
            raise ValueError("seek target before match start")

        i = pos
        while self.entries[i].segment_type != 0x01:  # walk back to nearest keyframe
            i -= 1
            if i < 0:
                raise ValueError("no keyframe precedes target timestamp")
        keyframe = self.entries[i]

        # Delta blocks between the keyframe and the target timestamp must be
        # replayed forward to land exactly on target_ms - this is the bounded
        # "step forward" cost that the keyframe interval controls.
        forward_blocks = [
            e for e in self.entries[i + 1:pos + 1]
            if e.segment_type == 0x02 and e.timestamp_ms <= target_ms
        ]
        return keyframe, forward_blocks

The index is published as a small JSON or protobuf manifest alongside the replay file, not kept only in a server-side database, precisely so a client can perform this binary search locally and then issue a single CDN range request for [keyframe.byte_offset, target_end_offset) instead of round-tripping to a stateful server for every scrub.

The Replay Server and Playback API

This component’s job is to hand a viewer everything they need to start watching a replay, and then get out of the way.

The instinct many engineers reach for is a stateful replay session: a server process that holds open a connection per viewer, tracks their current playback position, and streams frames to them, similar to how a live spectator feed might work. That model does not survive contact with 8 million playback sessions a day, because it means server capacity scales linearly with concurrent viewers, and a viral highlight clip would need to spin up an enormous number of simultaneous sessions on demand. Instead, the Replay Server’s job on the hot path is reduced to one cheap call per session: return the manifest (replay file location plus the seek index) for the requested match.

# Replay Server: playback session bootstrap response
# Demonstrates: the one call per session that touches the origin at all
playback_manifest:
  match_id: 8823140192
  replay_url: "https://cdn.example-games.com/replays/2026/07/02/8823140192.rpl"
  duration_ms: 1198500
  codec: "zstd-dict-v3"
  keyframe_interval_ms: 15000
  seek_index_url: "https://cdn.example-games.com/replays/2026/07/02/8823140192.idx"
  storage_tier: "hot"
  expires_at: "2026-08-01T00:00:00Z"

Once the client has this manifest, every subsequent scrub is a direct HTTP range request from the client to the CDN, computed locally using the SeekIndex.plan_seek logic from the previous section. The Replay Server never sees those requests. This is the same shape as the split between the ingest gateway and the fully self-contained candidate lookup in a search system: one cheap bootstrap call, then a self-sufficient client-driven path for everything after.

CDN Delivery of Finished Replays

This component’s job is to serve the actual replay bytes to millions of viewers a day using origin capacity that stays flat no matter how popular any single replay gets.

The naive design serves replay files straight from the blob store or the Replay Server on every request. That fails the same way any single-origin design fails under fan-out: one popular replay (a tournament grand final, a viral clip) can draw an enormous share of daily traffic, and origin bandwidth does not scale the way CDN edge bandwidth does. Once a replay is finalized, we push it, along with its seek index manifest, to CDN storage immediately, and every subsequent read, whether it is the first ten seconds of playback or a scrub to the middle of the match, is served as an HTTP range request against the CDN edge.

# CDN push configuration for a newly finalized replay
# Demonstrates: cache behavior tuned for range-request-heavy replay access
cdn_config:
  origin_shield: true            # collapses concurrent cold-cache fetches into one origin call
  cache_control: "public, max-age=2592000, immutable"  # finalized replays never change
  accept_ranges: true
  edge_ttl_seconds: 2592000       # 30 days, matches the hot+warm retention window
  prewarm_on_publish: false       # only prewarmed explicitly for flagged/tournament matches
  prewarm_trigger: "tournament_flag OR projected_view_count > 50000"

This mirrors exactly what HLS and DASH do for video streaming: a manifest lists segment or byte boundaries aligned to keyframes, and a player seeks by fetching the CDN-cached chunk nearest the target time, never asking the origin to transcode or reassemble anything on demand. Our seek index plays the same role as an HLS media playlist, just for a structured event log instead of encoded video frames.

Storage Lifecycle: Hot, Warm, Cold, and Expiry

This component’s job is to move a replay through a sequence of storage tiers as it ages, so that the enormous majority of storage cost is spent on the small fraction of replays anyone still watches.

The instinct to avoid is treating “store forever” and “delete after N days” as the only two options. Most replays are watched heavily in the first week after a match (players reviewing their own games, friends checking a highlight) and then almost never again, except for a small minority: tournament matches, matches flagged for anti-cheat review, or a clip that unexpectedly goes viral months later. A single retention policy either wastes money keeping cold data at hot-tier cost, or deletes something that turns out to matter.

-- Lifecycle transition query: find matches ready to move from hot to warm tier
-- Demonstrates: date-partitioned, flag-aware lifecycle selection
SELECT match_id, blob_key
FROM replay_matches
WHERE storage_tier = 'hot'
  AND started_at < NOW() - INTERVAL '7 days'
  AND flagged = FALSE
LIMIT 10000;

-- Matches flagged for tournament or anti-cheat hold are explicitly exempt
-- from the standard retention timer and must be moved to a long-lived
-- cold archive tier instead of the normal 30-day deletion path.
SELECT match_id, blob_key
FROM replay_matches
WHERE storage_tier IN ('hot', 'warm')
  AND started_at < NOW() - INTERVAL '30 days'
  AND flagged = TRUE
  AND storage_tier != 'cold';

We keep hot-tier replays (0 to 7 days old) at full fidelity, the original 15-second keyframe interval. Past 7 days, a background job re-encodes the replay with a sparser 60-second keyframe interval before moving it to the warm tier, trading a little seek latency (more delta blocks to step through per scrub) for roughly a 4x reduction in size, since most warm-tier replays are scrubbed far less often than fresh ones. Ordinary, unflagged matches are deleted entirely after 30 days. Flagged matches (tournament, reported, under anti-cheat review) move to a cold archive tier at the same reduced fidelity and are retained for a full year.

Data Model

The schema separates three concerns: match-level metadata and lifecycle state, the physical byte layout of each replay file, and the seek index entries published to clients.

-- Replay match metadata: one row per match, drives lifecycle transitions
CREATE TABLE replay_matches (
    match_id      BIGINT       PRIMARY KEY,
    game_mode     TEXT         NOT NULL,
    region        TEXT         NOT NULL,
    started_at    TIMESTAMPTZ  NOT NULL,
    ended_at      TIMESTAMPTZ,
    duration_ms   INTEGER,
    player_count  SMALLINT     NOT NULL,
    status        TEXT         NOT NULL DEFAULT 'recording',  -- recording | finalized | archived | deleted
    storage_tier  TEXT         NOT NULL DEFAULT 'hot',        -- hot | warm | cold | deleted
    blob_key      TEXT,                                       -- set once finalized
    flagged       BOOLEAN      NOT NULL DEFAULT FALSE,        -- tournament / reported / anti-cheat hold
    created_at    TIMESTAMPTZ  NOT NULL DEFAULT NOW()
) PARTITION BY RANGE (started_at);

CREATE INDEX idx_replay_matches_status ON replay_matches (status) WHERE status != 'deleted';
CREATE INDEX idx_replay_matches_lifecycle ON replay_matches (storage_tier, started_at) WHERE flagged = FALSE;

-- Replay segments: byte-layout metadata for each keyframe / delta block written to disk
CREATE TABLE replay_segments (
    match_id      BIGINT    NOT NULL REFERENCES replay_matches(match_id),
    segment_seq   INTEGER   NOT NULL,
    timestamp_ms  INTEGER   NOT NULL,
    byte_offset   BIGINT    NOT NULL,
    byte_length   INTEGER   NOT NULL,
    segment_type  SMALLINT  NOT NULL,           -- 1 keyframe, 2 delta block
    codec         TEXT      NOT NULL DEFAULT 'zstd-dict-v3',
    PRIMARY KEY (match_id, segment_seq)
) PARTITION BY RANGE (match_id);

-- Seek index entries: the compact timestamp -> byte offset mapping,
-- published verbatim as the client-facing manifest
CREATE TABLE seek_index_entries (
    match_id      BIGINT    NOT NULL REFERENCES replay_matches(match_id),
    timestamp_ms  INTEGER   NOT NULL,
    byte_offset   BIGINT    NOT NULL,
    segment_type  SMALLINT  NOT NULL,
    PRIMARY KEY (match_id, timestamp_ms)
);

CREATE INDEX idx_seek_index_match ON seek_index_entries (match_id, timestamp_ms);

replay_matches is partitioned by started_at in daily ranges, which is what makes the lifecycle jobs cheap: moving a day’s worth of matches from hot to warm, or dropping an entire day’s worth of expired, unflagged matches, becomes a partition-level operation rather than millions of individual row deletes. replay_segments is partitioned by match_id range because every query against it is scoped to a single match, never a time range, so match-based partitioning keeps a segment lookup local to one partition. seek_index_entries mirrors the manifest we actually publish to clients, and exists in the database mainly so the Replay Server can regenerate a manifest on demand (for example, after a lifecycle re-encode changes byte offsets) without re-parsing the raw replay file.

A replay moves through six states over its life: recording (events landing in the ring buffer, nothing durable yet), flushed (event batches durably written by the Replay File Writer), finalized (the complete keyframe-delta file assembled at match end), indexed (the seek index built and published as a manifest), hot (served at full fidelity from CDN for 7 days), and finally either warm/cold (re-encoded at lower fidelity, retained per the flag-aware policy) or deleted.

Key Algorithms and Protocols

Seek Index Lookup

Covered in the Seek Index component above (SeekIndex.plan_seek), the core operation is a binary search over a sorted array of (timestamp_ms, byte_offset, segment_type) entries, which runs in O(log n) where n is the number of segments in the replay, typically in the low thousands for a twenty-minute match. Once the nearest preceding keyframe is located, the “forward decode” step (replaying delta blocks between the keyframe and the target timestamp) is bounded by the keyframe interval and the delta block window, not by how far into the match the target timestamp falls. This is the property that makes seek latency roughly constant regardless of whether the viewer scrubs to second 10 or minute 19 of the same match.

Keyframe Interval Selection

Choosing KEYFRAME_INTERVAL_MS is a direct tradeoff between storage size and worst-case seek decode cost, and the right value differs by game mode.

# Keyframe interval tuning: sweep candidate intervals against storage and
# worst-case forward-decode cost to find an acceptable operating point
from dataclasses import dataclass

@dataclass
class MatchProfile:
    avg_keyframe_bytes: int      # size of one full state snapshot for this game mode
    avg_delta_bytes_per_sec: int # size of the delta stream per second of match time

def estimate_storage_per_match(profile: MatchProfile, interval_ms: int, duration_ms: int) -> int:
    num_keyframes = duration_ms // interval_ms
    keyframe_bytes = num_keyframes * profile.avg_keyframe_bytes
    delta_bytes = (duration_ms / 1000) * profile.avg_delta_bytes_per_sec
    return int(keyframe_bytes + delta_bytes)

def estimate_worst_case_forward_decode_ms(interval_ms: int, delta_decode_ms_per_sec: float) -> float:
    # Worst case: seek target lands just before the next keyframe, so the
    # client decodes almost a full interval's worth of delta blocks forward
    return (interval_ms / 1000) * delta_decode_ms_per_sec

def choose_interval(profile: MatchProfile, duration_ms: int, max_seek_decode_ms: float) -> int:
    # Walk from short to long intervals, pick the longest one that still
    # keeps worst-case forward decode under the latency budget
    best = 1_000
    for interval_ms in range(1_000, 60_000, 1_000):
        decode_cost = estimate_worst_case_forward_decode_ms(interval_ms, delta_decode_ms_per_sec=0.6)
        if decode_cost <= max_seek_decode_ms:
            best = interval_ms
        else:
            break
    return best

A fast-paced mode with forty players and constant combat has a larger, more expensive delta stream per second than a slower, smaller-lobby mode, so it benefits from a shorter keyframe interval even though that costs more storage per match; a slower mode can tolerate a longer interval and save on storage without pushing worst-case seek decode past budget. We chose 15 seconds empirically for the flagship mode described in this post, the point where storage growth from shortening the interval further stopped buying a meaningful decode-latency improvement.

Compression Strategy

Delta events are individually tiny, tens of bytes, which means compressing them one at a time has terrible overhead relative to payload size. Compressing an entire match’s delta stream as one large block gets a much better ratio, but then seeking near the end of that block requires decompressing almost the whole thing, which directly fights the seek-latency goal. The fix is to batch deltas into small, independently compressed blocks, short enough that decompressing one is cheap, but long enough that compression still pays for itself.

# Compression strategy: batch deltas into small zstd blocks with a shared
# trained dictionary, keeping each block small enough to preserve seek granularity
import zstandard as zstd

DELTA_BLOCK_WINDOW_MS = 1_000     # one compressed block per ~1 second of deltas
KEYFRAME_DICT_ID = "replay-keyframe-v3"
DELTA_DICT_ID = "replay-delta-v3"

def compress_delta_block(events: bytes, dictionary: zstd.ZstdCompressionDict) -> bytes:
    # A 1-second window means a seek only ever needs to decompress at most
    # a handful of ~1-second blocks between the keyframe and the target,
    # instead of an entire match's worth of deltas
    cctx = zstd.ZstdCompressor(level=12, dict_data=dictionary)
    return cctx.compress(events)

def compress_keyframe(full_state: bytes, dictionary: zstd.ZstdCompressionDict) -> bytes:
    # Keyframes are large and highly repetitive (entity tables, fixed schema
    # fields) so a higher compression level pays for itself; keyframes are
    # requested far less often than delta blocks so the extra CPU is cheap
    cctx = zstd.ZstdCompressor(level=19, dict_data=dictionary)
    return cctx.compress(full_state)

The dictionaries are trained offline on a sample of recorded matches and versioned (replay-delta-v3), because both keyframes and delta events share a lot of structure across matches (the same entity schema, the same event type distribution), and a shared dictionary captures that redundancy far better than compressing each block cold. Retraining a dictionary requires bumping the version so old replays keep referencing the dictionary they were actually compressed against.

Scaling and Performance

The system scales along two independent axes: more ingest and encoding capacity to absorb more concurrent live matches, and more CDN edge capacity to absorb more playback traffic. Because encoding and indexing are per-match, stateless operations and playback is entirely CDN-served after the initial manifest fetch, both axes scale horizontally without any cross-match coordination.

Capacity Estimation:

Given:
  - 20,000,000 matches/day, avg duration 1,200 sec (20 min)
  - ~50 delta events/sec/match aggregate, ~24 bytes/event encoded
  - keyframe every 15 sec, ~200KB per compressed keyframe
  - 8,000,000 replay playback sessions/day, ~15 seek/range requests/session

Per-match storage (hot tier, full fidelity):
  Delta stream: 50 events/sec * 1,200 sec * 24 bytes = 1.44 MB raw
    after 1-sec block compression (~3x): ~0.48 MB
  Keyframes: (1,200 sec / 15 sec) * 200 KB = 80 * 200KB = 16 MB
  Total: ~16.5 MB/match

Daily new storage (hot tier):
  20,000,000 matches * 16.5 MB ~= 330 TB/day

Storage by tier:
  Hot (0-7 days, full fidelity): 330 TB * 7 = ~2.3 PB
  Warm (8-30 days, re-encoded at ~4x smaller): (330 TB / 4) * 23 = ~1.9 PB
  Cold (flagged matches only, ~1.5% of matches, 1 year retention):
    20,000,000 * 0.015 * (16.5 MB / 4) ~= 1.24 TB/day -> ~452 TB/year

Playback request volume:
  8,000,000 sessions/day * 15 requests/session = 120,000,000 range requests/day
  Average: ~1,390 requests/sec, peak (~5x during evening hours): ~6,900 requests/sec

Origin load (manifest fetches + CDN cache misses only):
  Manifest fetches: 8,000,000/day ~= 93/sec average
  Cache misses (3% of range requests): 3,600,000/day ~= 42/sec average

Bandwidth:
  Avg bytes served per session (partial, range-request driven): ~6 MB
  8,000,000 sessions * 6 MB = 48 TB/day served, 97%+ from CDN edge

Reads dominate writes by a wide margin, but not because playback is more frequent than recording on a per-match basis, it is because a single finalized replay fans out to potentially thousands or millions of viewers, while it is only ever written once. The real hot spot is not read/write ratio, it is skew: a viral clip or a tournament final draws a disproportionate share of playback traffic to one replay file. We handle this the same way any CDN-backed system does, by prewarming edge caches ahead of known high-traffic events (tournament matches, per the prewarm_trigger condition earlier) and relying on origin shielding to collapse redundant cold-cache fetches into a single origin request rather than one per concurrent viewer.

Failure Modes and Recovery

Failure	Detection	Impact	Recovery
Game server crashes mid-match before finalization	Watchdog sees no finalize signal within the expected match-end window	Replay ends abruptly at the last flushed batch instead of the true match end	Recorder flushes every few seconds, not only at match end, so at most a few seconds of tail events are lost; the writer finalizes a valid, playable (truncated) replay from whatever was durably flushed
Keyframe segment corrupted on disk (bit rot, partial write)	Per-segment checksum mismatch on read	Seeking near that keyframe fails to reconstruct correct state	Seek planner falls back to the next older valid keyframe and replays forward through more delta blocks; multi-AZ replicated storage makes this rare, and a corrupted replica triggers a re-fetch from a healthy one
Seek index not yet built when a viewer requests playback	Manifest request to the Replay Server returns “index pending” instead of a manifest	Viewer cannot scrub, only sequential playback from the start is possible	Client falls back to a slow linear-scan path using raw segment headers; index builder retries with backoff, and lag past the 30-second finalization SLA pages the on-call engineer
CDN cache miss storm on a newly viral replay	Origin request rate for one match_id spikes far above baseline	Origin (blob store, Replay Server) risks overload serving the same byte ranges repeatedly	Origin shielding collapses concurrent cache-miss fetches for the same range into a single origin request; known high-profile matches are prewarmed at the edge ahead of expected demand
Lifecycle job deletes a replay still under anti-cheat or tournament hold	Deletion job re-checks flagged immediately before issuing the delete, not only at scheduling time	Loss of evidence needed for a ban appeal or match review	flagged is a hard gate re-read at delete time; flagged matches are routed to the cold archive path and explicitly excluded from every deletion query
Clock skew between the game server and the ingest pipeline	Sequence numbers and tick-based timestamps diverge from wall-clock arrival time beyond tolerance	Seek index timestamps could misalign, causing a scrub to land near, but not at, the correct moment	The system indexes by monotonic, match-relative tick_ms, never wall-clock time, so seek targets are immune to NTP adjustments or clock drift on any single machine

Comparison of Approaches

Approach	Latency	Complexity	Failure Mode	Best Fit
Full state snapshot on every tick	Seek is instant, already at the right position	Low	Storage cost explodes; a 20-minute match at even 10 snapshots/sec becomes enormous	Very short debug captures, not production-scale replay
Pure event log, no keyframes	Seek cost grows linearly with position in the match, unbounded worst case	Low	Late-match scrubs become unusably slow as the match gets longer	Very short sessions, or systems where only rewind-from-start matters
Keyframe + delta hybrid with a seek index (this design)	Bounded, sub-second seek anywhere in the match	High (index building, keyframe interval tuning, compression block sizing)	Misconfigured keyframe interval degrades either storage or worst-case seek latency	Long-form matches with millions of daily scrubbable playback views
Full video re-encoding of the match (rendered pixels)	Excellent, standard video-player seeking via GOP-aligned keyframes	Very high (rendering pipeline, transcoding infra, huge storage)	Storage and bandwidth costs an order of magnitude higher; loses interactive camera control and stat overlays	Broadcast VOD clips, not general player-facing replay tooling with free camera or UI overlays
Stateful per-viewer replay session on the origin	Good once connected, but compute cost scales with concurrent viewers	Very high (needs a live simulation instance per session)	Thundering herd of compute sessions on a popular replay; does not scale to millions of daily views	Live in-progress spectating, a related but different problem than post-match replay

For player-facing post-match replay at this scale, the keyframe-and-delta hybrid backed by a client-driven seek index is the right call: it bounds decode cost regardless of match length, keeps the file small enough that a full match replay is a few megabytes rather than hundreds, and lets a CDN carry nearly all playback traffic instead of the origin. Full video re-encoding solves a genuinely different problem, producing a broadcast-ready artifact, and would be the right choice for auto-generated highlight clips shared outside the game client, not for the interactive, camera-controllable replay viewer this system targets. Mixing the two into one system would mean paying full rendering and transcoding cost on every single match just to support a feature (free camera scrubbing) that a compact event log already delivers far more cheaply.

Key Takeaways

• Keyframe and delta encoding is what makes sub-second scrubbing possible: seeking never requires decoding more than one keyframe interval’s worth of delta blocks, bounding decode cost regardless of match length.
• The seek index is a manifest published to the client, not a server-side lookup table: this is what lets a scrub become a direct CDN range request instead of a round trip to a stateful server on every seek.
• Event recording must never touch the simulation tick: the ring buffer plus async flush pattern means the live game server drops old, unflushed events under backpressure rather than ever blocking on I/O.
• Compression block size is a seek-granularity knob, not just a storage knob: batching deltas into windows that are too large trades away seek latency for a marginally better compression ratio.
• Keyframe interval is a measured tradeoff, not a fixed constant: shorter intervals cost more storage and buy a lower worst-case seek cost, and the right interval differs by game mode.
• Reads dominate writes by an enormous margin: one finalized replay file can fan out to millions of playback sessions, which is why CDN offload matters far more than write throughput.
• Storage lifecycle needs an explicit, re-checked hold flag, not just a retention timer: tournament and anti-cheat-flagged replays must be exempt from the same timer that deletes ordinary matches after 30 days.
• Monotonic, tick-based timestamps make seeking immune to clock skew: indexing by match-relative tick count rather than wall-clock time avoids an entire class of seek-misalignment bugs.

The counter-intuitive lesson is that capturing the data is the easy part, it is a straightforward buffering problem with a well-known answer. The hard part is designing a file format and an index that make “jump anywhere instantly” cheap without storing every tick of full state, which really means picking, deliberately and per game mode, exactly how much replay work you are willing to defer from write time to read time. Every keyframe interval, every compression block boundary, and every CDN cache TTL in this design is a version of that same one tradeoff.

Frequently Asked Questions

Q: Why not just record full video of the match and let viewers scrub it like a normal video file?

A: Video seeking already solves scrubbing well through GOP-aligned keyframes, but it requires a full rendering and transcoding pipeline, costs far more in storage and bandwidth per match, and throws away the interactivity players actually want from a game replay: a free camera, stat overlays, jumping between player perspectives. Event-based replay keeps the file small enough to ship a whole match in a few megabytes and lets the client render however it wants from the recorded state.

Q: Why not snapshot the full game state on every single tick instead of encoding deltas between keyframes?

A: Snapshotting every tick makes every seek instant, but at 128 ticks/sec for a twenty-minute match with forty entities, the storage cost becomes enormous, easily hundreds of megabytes per match instead of the roughly 16.5MB this design achieves. The keyframe-and-delta hybrid accepts a small, bounded amount of forward decode work per seek in exchange for an order-of-magnitude smaller file.

Q: Why does the seek index live in a manifest fetched once by the client instead of only in a database the Replay Server queries on every scrub?

A: Querying a server on every scrub would make the Replay Server’s capacity the ceiling on how many people can watch replays simultaneously, and it defeats the entire point of pushing replay bytes to a CDN. Publishing the index as a small file alongside the replay lets the client compute its own byte ranges and issue direct CDN range requests, so the origin is touched exactly once per playback session, not once per seek.

Q: Why does the game server buffer events in memory and flush asynchronously instead of writing each event straight to durable storage?

A: A synchronous write on the simulation thread, even a fast one, adds latency and jitter to the tick that is also computing the next frame of live gameplay. Buffering in a non-blocking ring buffer and flushing on a separate thread keeps event capture effectively free from the simulation loop’s perspective, at the cost of a small, bounded window of potential data loss if the process crashes between flushes.

Q: How do you decide the keyframe interval, and does it differ by game mode?

A: It is chosen by sweeping candidate intervals against two measurable costs, per-match storage and worst-case forward-decode time, and picking the longest interval that still keeps worst-case decode under the seek latency budget. Faster-paced modes with a larger delta stream per second justify a shorter interval even at higher storage cost, while slower modes can use a longer interval and save on storage without missing the latency target.

Q: Why not keep a stateful replay server process per active viewer instead of a stateless CDN-served file?

A: A stateful session per viewer means compute capacity scales linearly with concurrent viewers, which collapses under a viral replay or a tournament final drawing hundreds of thousands of simultaneous viewers. A stateless, CDN-served file with a client-computed seek index scales with CDN edge capacity instead, which is built for exactly this kind of fan-out. A stateful per-viewer model is the right shape for live in-progress spectating, a genuinely different problem, but not for post-match replay.

Interview Questions

Q: Design the seek index structure for a replay system that must support sub-second scrubbing to any point in a twenty-minute match. How would you approach this?

Expected depth: Discuss why linear scanning of segment headers does not scale to the target request volume, how binary search over a sorted timestamp-to-offset array gives O(log n) lookup, and why the index needs to be published as a small manifest alongside the file rather than queried from a server on every request. Cover the keyframe-plus-forward-decode cost model and why it bounds worst-case seek latency independent of match length.

Q: How would you choose the keyframe interval for a new game mode, and what happens if you get it wrong in either direction?

Expected depth: Cover the storage-versus-decode-latency tradeoff, how to measure both sides empirically (average keyframe size, delta stream rate, worst-case forward-decode time), and why the right interval differs by how event-dense the game mode is. Discuss the failure modes at each extreme: too short wastes storage, too long blows the seek latency budget on the worst-case query just before the next keyframe.

Q: A tournament match goes viral and draws five hundred thousand replay views in an hour. Walk through what happens end to end.

Expected depth: Discuss CDN edge cache behavior under a traffic spike, the role of origin shielding in collapsing redundant cache-miss fetches, and why prewarming the edge ahead of known high-profile matches (rather than reacting after the spike starts) is the practical mitigation. Touch on why the Replay Server’s load stays flat (one manifest fetch per new session) even as scrub volume for that one match explodes.

Q: How does the system avoid adding latency to the live game server’s tick loop while still durably capturing every event?

Expected depth: Explain the ring buffer plus async flush pattern, why the simulation thread must never perform blocking I/O, the tradeoff between buffer size and data-loss window on crash, and how backpressure is handled by dropping the oldest unflushed events rather than blocking the tick that is generating them.

Q: Design the storage lifecycle for replays: what tiers exist, how does data move between them, and how do you guarantee you never delete something under legal or anti-cheat hold?

Expected depth: Cover the hot, warm, cold, and deleted tiers, the fidelity tradeoff of re-encoding at a longer keyframe interval as data ages, and date-partitioned lifecycle jobs that operate on whole partitions rather than individual rows. Emphasize that the flag check must be re-read immediately before every deletion, not cached from when the job was scheduled, since a flag can be set after a deletion was already queued.