Build a Game Replay Storage and Scrubbing System


data-engineering scalability performance

System Design Deep Dive

Game Replay Storage and Scrubbing

Every match event captured, any second scrubbable in under a second, served off a CDN instead of the game servers.

⏱ 17 min read 📐 Advanced 🗄️ Data Engineering

A thousand-page reference manual never makes you start on page one to find page four hundred. It has a table of contents that points you at the nearest chapter start, and once you land there you skim forward a few paragraphs to the exact sentence you wanted. Nobody rereads the whole book to jump to the middle of it.

Scrubbing a game replay to the fourteen-minute mark needs exactly the same trick. A finished match is really a long sequence of tiny state changes: a player moves, a weapon fires, a shield breaks, a round ends. Play all of those back in order from the beginning and you get a perfect reconstruction of the match, but if a viewer wants to jump straight to the game-winning play at minute fourteen, replaying thirteen minutes and fifty-nine seconds of prior events just to get there is not an option. We need something like that table of contents: a way to land near the right moment instantly, then step forward only a handful of small updates to reach the exact instant the viewer asked for.

The system has to do this at a scale that makes the naive answers fall apart fast. Every match on the platform generates a continuous stream of gameplay events that has to be captured durably, without adding latency to the live simulation loop that is also, at that exact moment, rendering the next frame for every player in the match. Once a match ends, its replay has to become scrubbable to any point in under a second, and it has to stay scrubbable for millions of playback sessions a day, spread across recent matches, viral highlight clips, and tournament VODs, without every one of those sessions hitting the same origin servers that also run live matches.

Three forces are in tension here. Capture has to be effectively free: recording an event can never slow down the tick that produced it. Storage has to be efficient: recording the full game state on every single tick would make storage costs balloon, but the format still has to support near-instant seeking, which pulls in the opposite direction. And delivery has to scale independently of origin capacity: millions of daily replay views cannot each be a fresh round trip to the servers that wrote the file. We need to solve for non-blocking event capture on a live game server, a file format that mixes full state snapshots with cheap incremental updates so seeking never means replaying from the start, and a delivery path that lets a CDN absorb almost all playback traffic, all three at once.

Requirements and Constraints

Functional Requirements

  • Capture every gameplay event (movement, actions, damage, state transitions) generated by a live match into a durable, ordered log
  • Assemble a finished match’s event log into a single replay file once the match ends
  • Let a viewer scrub to any timestamp in a completed replay and begin playback from that exact point
  • Support common playback controls built on scrubbing: play, pause, fast-forward, rewind, and jump-to-event (for example, jump to the next kill or the start of the next round)
  • Serve replay playback to spectators, VOD viewers, and broadcast tooling without those requests reaching the live game servers
  • Compress recorded event and snapshot data for storage efficiency while preserving fast, granular seeking
  • Move replays through a storage lifecycle: fresh and fast to access right after the match, cheaper and slower after that, archived indefinitely only when flagged, deleted otherwise

Non-Functional Requirements

  • Recording overhead: event capture on the game server adds no more than 0.5ms to any single simulation tick, and never blocks the tick loop waiting on I/O
  • Scale: roughly 20 million matches/day, average match length 20 minutes (1,200 seconds), average 40 players/match, an aggregate delta-event rate of about 50 events/sec per match
  • Seek latency: p99 under 1 second for a cold (uncached) seek anywhere in a finished replay; under 150ms for a seek served from a warm CDN cache
  • Finalization latency: a replay becomes playable within 30 seconds of the match ending
  • Playback scale: roughly 8 million replay playback sessions/day, each issuing on average 15 seek or range requests, for about 120 million total range requests/day
  • Origin protection: at least 97% of replay playback bytes served from CDN edge cache, never touching the origin replay servers
  • Durability: once a replay is finalized, 99.999999999% durability (S3-class) until its scheduled deletion or archival
  • Availability: 99.9% availability for replay playback; the recording path fails toward “record less, never block the game” rather than dropping frames of live gameplay

Constraints and Assumptions

  • Out of scope: live in-progress spectating (watching a match as it happens). This system builds post-match replay only; live spectating is a related but separate real-time streaming problem
  • We assume the game client can deterministically (or near-deterministically, with periodic full-state correction) reconstruct what a match looked like from its recorded events; guaranteeing that reconstruction is a game-engine responsibility, not something this storage system provides
  • Voice chat and rendered video capture are out of scope. This is event-based replay data, not a video recording
  • Anti-cheat review and moderation are downstream consumers of the raw event log; we build the storage and retrieval layer they read from, not the review tooling itself
  • We assume a single authoritative region per match (the region the game server ran in); cross-region replay access is served through the CDN, not a cross-region write path

High-Level Architecture

The system has five major components: the Event Recorder running inside the live game server process, the Replay File Writer that encodes captured events into a keyframe-and-delta file format, the Seek Index Builder that produces a timestamp-to-byte-offset manifest for each finished replay, the Replay Server that handles session start and metadata, and the CDN Delivery Layer that serves the actual replay bytes to viewers.

Game replay storage and scrubbing system architecture overview

During a live match, the Event Recorder sits inside the game server’s process and buffers every event (a move, a cast, a damage tick) into an in-memory ring buffer, without ever making the simulation tick wait on disk or network I/O. A separate flush goroutine drains that buffer on a fixed interval and ships batches to the Replay File Writer, which is what actually decides, per batch, whether to emit a full keyframe (a complete snapshot of match state) or a small delta block (the incremental changes since the last snapshot). When the match ends, the writer finalizes the file, and the Seek Index Builder walks the finished file to produce a compact manifest mapping playback timestamps to byte offsets. That manifest, plus the replay file itself, gets pushed to the CDN. From that point forward, a viewer’s player client fetches the manifest once from the Replay Server at session start, then issues HTTP range requests directly against the CDN edge to scrub anywhere in the match, without the Replay Server or the original game server being involved at all.

That last point is the crux of the whole design: the seek index is not a database the Replay Server queries on every scrub, it is a small file published alongside the replay itself, so that after the first request of a session, the entire playback experience runs against CDN-cached bytes.

Key Insight

The system splits cleanly into a write path that only ever runs once per match (recording, encoding, indexing) and a read path that runs millions of times per day (playback). Making the write path do anything more than the minimum required to produce a correct, seekable file would risk adding latency to live gameplay. Making the read path depend on a stateful server for every scrub would mean the Replay Server's capacity, not the CDN's, becomes the ceiling on how many people can watch replays. Neither path is allowed to become the other's bottleneck.

Event Recording on the Live Game Server

This component’s job is to get every gameplay event out of the simulation loop and into a durable pipeline without the simulation loop ever noticing it happened.

The naive instinct is to write each event straight to a database or a network socket the moment it occurs, because that is the simplest code to write and it means nothing is ever lost. The problem is that the simulation tick that produces these events is also the tick computing physics, hit detection, and the next frame every player in the match is about to see. Any synchronous I/O call on that thread, even a fast one, adds jitter to gameplay, and jitter on a 60Hz or 128-tick server is something players feel immediately. The fix is to make event capture a pure, in-memory append with zero I/O, and move all I/O to a separate thread that the simulation loop never waits on.

// Game server: non-blocking event capture into an in-memory ring buffer
// Demonstrates: how the simulation tick loop hands off events without blocking on I/O
package recorder

import (
	"sync"
	"time"
)

type EventType uint8

const (
	EventMove EventType = iota
	EventDamage
	EventAbilityCast
	EventStateChange
)

type GameEvent struct {
	SeqNo    uint32 // monotonic, per-match, tick-based - never wall clock
	TickMs   uint32 // milliseconds since match start
	Type     EventType
	EntityID uint32
	Payload  []byte // small, event-type-specific encoded fields
}

// RingBuffer is a fixed-capacity, single-writer, single-reader buffer.
// The simulation goroutine only ever calls Push, which never allocates
// and never blocks - if the buffer is full, the oldest unflushed event
// is dropped and a counter is incremented instead of stalling the tick.
type RingBuffer struct {
	mu      sync.Mutex
	buf     []GameEvent
	head    int
	size    int
	dropped uint64
}

func NewRingBuffer(capacity int) *RingBuffer {
	return &RingBuffer{buf: make([]GameEvent, capacity)}
}

func (r *RingBuffer) Push(e GameEvent) {
	r.mu.Lock()
	defer r.mu.Unlock()
	idx := (r.head + r.size) % len(r.buf)
	if r.size == len(r.buf) {
		r.head = (r.head + 1) % len(r.buf) // overwrite oldest, count the drop
		r.dropped++
	} else {
		r.size++
	}
	r.buf[idx] = e
}

// DrainBatch is called by a separate flush goroutine on a fixed interval,
// never by the simulation thread itself.
func (r *RingBuffer) DrainBatch(max int) []GameEvent {
	r.mu.Lock()
	defer r.mu.Unlock()
	n := r.size
	if n > max {
		n = max
	}
	batch := make([]GameEvent, n)
	for i := 0; i < n; i++ {
		batch[i] = r.buf[(r.head+i)%len(r.buf)]
	}
	r.head = (r.head + n) % len(r.buf)
	r.size -= n
	return batch
}

// StartFlushLoop runs on its own goroutine, off the simulation tick,
// and ships batches to the ingest pipeline every flushInterval.
func StartFlushLoop(rb *RingBuffer, flushInterval time.Duration, ship func([]GameEvent) error) {
	ticker := time.NewTicker(flushInterval)
	go func() {
		for range ticker.C {
			batch := rb.DrainBatch(2000)
			if len(batch) == 0 {
				continue
			}
			if err := ship(batch); err != nil {
				// Ship failures never propagate back to the simulation loop;
				// the batch is re-queued for retry with backoff by the caller.
				continue
			}
		}
	}()
}

Think of the ring buffer like a stenographer taking shorthand notes during a live meeting: they write fast, cheap symbols in the moment and never stop to look anything up, leaving the actual transcription to happen later, off to the side, without disrupting the meeting. If the meeting talks faster than the stenographer can keep up, a real stenographer drops the least important filler, not the whole meeting; our ring buffer does the same by discarding the oldest unflushed events under sustained backpressure rather than blocking Push.

What would break if we simplified this to a single mutex-protected queue with unbounded growth instead of a fixed-size ring buffer? Under a burst (a team fight with forty players simultaneously casting abilities), the queue could grow without bound, and a slow flush consumer would let memory balloon on the game server process that is also running the live simulation, eventually causing the kind of GC pause or OOM that is far worse for gameplay than losing a few seconds of replay fidelity.

Premium Content

Unlock the full article along with everything else in the archive — all in one place.

In-depth analysis Expert insights Full archive access
Unlock Full Article