Picture the final hour of a Fortnite Championship Series event. Ten million players are in active matches. Every elimination, every placement bonus, every storm survival second ticks upward on their score. The moment a match ends and scores are submitted, players sprint to the leaderboard screen to see whether they cracked the top 100 - and whether the player who was ranked 97th an hour ago is now ranked 104th and out of the prize pool. This is not a slow read-and-display problem. This is a live, high-velocity write problem where the read surface must stay instantaneous under crushing load.

The numbers are unforgiving. At peak submission windows - the last 30 minutes of a tournament round when thousands of matches resolve simultaneously - score updates arrive at 50,000 per second. Each update must find the player’s current entry, compare it to the incoming score, apply the update if it is higher, and recompute that player’s rank relative to all 10 million other players. The top-100 leaderboard is requested by players, spectators, and broadcast overlays at roughly 200,000 reads per second. Each of those reads must complete in under 10ms.

The personal rank query is a separate beast. A player finishing a match immediately types “where am I?” They want their rank among 10 million peers, not just whether they made the top 100. This query - “given my score, how many players have a strictly higher score?” - looks trivial but is surprisingly expensive at 10M entries. Doing a full table scan on every personal rank query would saturate any database that exists. The answer must come back in under 5ms.

We need to solve for high-velocity score ingest without write amplification, sub-10ms top-N serving with minimal cache invalidation churn, and sub-5ms personal rank estimation that degrades gracefully to approximation rather than failing under load simultaneously.

Requirements and Constraints

Functional Requirements

• Accept score updates for any of the 10 million active players in real time during a tournament window
• Serve the live top-100 leaderboard globally (rank, player ID, score, display name)
• Answer personal rank queries: “what rank is player X right now?”
• Enforce score monotonicity: only accept a new score if it is strictly greater than the player’s current score (scores can only increase within a tournament)
• Support tie-breaking by insertion timestamp: players with identical scores are ranked by who achieved that score first

Non-Functional Requirements

• Write throughput: 50,000 score updates per second sustained, 80,000 at peak (match-end bursts)
• Top-100 read latency: under 10ms at P99 globally
• Personal rank latency: under 5ms at P99
• Availability: 99.99% during the active tournament window (4-8 hours)
• Consistency: top-100 list reflects all score updates within 1 second of submission; personal rank may be approximate within +/- 500 positions at the 10M scale
• Storage: 10M player entries with score + metadata; approximately 400MB per sorted set

Constraints and Assumptions

• The tournament has a defined start and end time; the leaderboard is read-heavy during the event and cold outside it
• Scores are non-negative integers, updated at most once every 10 seconds per player (game engine rate limit)
• Score updates may arrive out of order due to network jitter; the system must apply only higher scores
• We accept rank approximation for personal rank queries beyond the top 10,000 positions
• The top-100 list is the globally authoritative number; it must be exact, not approximated

High-Level Architecture

The system splits into a write path that ingests score updates and a read path that serves rankings.

Score updates arrive from Game Servers via a REST or gRPC endpoint at the Score Ingestion Service. Rather than writing directly to the leaderboard store, the ingestion service enqueues each update to Kafka (partitioned by player_id to preserve per-player ordering). This decouples the game server from the leaderboard’s write speed and provides a durable buffer during burst windows.

The Leaderboard Compute Service consumes from Kafka, applies score-monotonicity checks, and writes to the Redis Sorted Set Cluster using ZADD NX and conditional Lua scripts. Redis sorted sets are the beating heart of this design: each member is a player ID, each score is the player’s tournament score, and Redis maintains the ordering automatically in O(log N) time per write.

On the read path, the Top-N API serves /leaderboard/top100 requests backed by a TTL-cached ZRANGE result. The cache sits behind a CDN layer that absorbs read amplification from millions of concurrent spectators. The Personal Rank Service answers individual rank queries by calling ZRANK against the Redis cluster and optionally approximating rank using a partition-merge strategy when the dataset is sharded.

Component Deep Dives

The Redis Sorted Set Layer

A Redis sorted set is a data structure where every member has an associated floating-point score and members are kept ordered by score at all times. This is not an afterthought feature - the skip list and hash table that implement it are specifically designed to make rank queries fast.

The three operations we rely on are:

• ZADD leaderboard {score} {player_id} - O(log N): add or update a player’s score. Combined with XX (update only existing) and GT (only if new score is greater), this enforces our monotonicity constraint atomically.
• ZRANK leaderboard {player_id} - O(log N): returns the zero-based rank of a player (0 = lowest score). We use ZREVRANK for descending ranking (0 = highest score).
• ZRANGE leaderboard 0 99 REV WITHSCORES - O(log N + K): returns the top-K entries with scores.

Memory sizing: each sorted set member requires approximately 40 bytes (8 bytes score as float64 + ~20 bytes player_id string + 12 bytes skip list overhead). At 10 million players: 10M * 40 bytes = 400MB. A single Redis node with 2GB memory holds this with headroom for the hash table index (roughly 1.5x the skip list size), totaling about 600MB. This fits on the smallest cloud Redis instance.

# Redis sorted set operations for leaderboard - core ZADD with score guard
# Demonstrates: monotonic score update using ZADD GT, ZREVRANK for descending rank

import redis

r = redis.Redis(host="leaderboard-redis", port=6379, decode_responses=True)

LEADERBOARD_KEY = "tournament:2026:leaderboard"

def update_score(player_id: str, new_score: float) -> bool:
    # ZADD with GT flag: only updates if new_score > current score
    # NX is NOT used here - we want updates, not inserts only
    # GT ensures monotonicity without a read-then-write race
    result = r.zadd(LEADERBOARD_KEY, {player_id: new_score}, gt=True)
    # result == 1 if the score was updated, 0 if existing was >= new_score
    return result == 1

def get_personal_rank(player_id: str) -> int | None:
    # ZREVRANK returns 0-based rank with highest score = rank 0
    rank = r.zrevrank(LEADERBOARD_KEY, player_id)
    if rank is None:
        return None
    return rank + 1  # convert to 1-based rank

def get_top_n(n: int = 100) -> list[dict]:
    # ZRANGE with REV=True returns highest scores first
    entries = r.zrange(LEADERBOARD_KEY, 0, n - 1, rev=True, withscores=True)
    return [
        {"rank": i + 1, "player_id": pid, "score": int(score)}
        for i, (pid, score) in enumerate(entries)
    ]

Score Update Pipeline

The Score Update Pipeline is built around three concerns: ordering, deduplication, and batching.

Think of it like a sports scoreboard operator at a stadium. Dozens of referees call in scores simultaneously over radio. If the operator wrote every individual call to the main board instantly, they would be overwhelmed during a scoring burst. Instead, they batch updates from each referee into 500ms windows and apply the highest confirmed score for each athlete.

Kafka partitioned by player_id gives us per-player ordering for free. All score updates for player X land in the same Kafka partition and are consumed in order. This means the Leaderboard Compute Service sees a player’s score history in chronological order, making monotonicity enforcement simple - track the last applied score per player in a local cache and skip events with lower scores before they even reach Redis.

# Leaderboard Compute Service: batched score update consumer
# Demonstrates: per-player score deduplication, batched ZADD pipeline

from kafka import KafkaConsumer
import redis
import json
import time
from collections import defaultdict

r = redis.Redis(host="leaderboard-redis", decode_responses=True)
LEADERBOARD_KEY = "tournament:2026:leaderboard"
BATCH_WINDOW_MS = 200  # 200ms batch window - low enough for 1s freshness SLO

consumer = KafkaConsumer(
    "score-updates",
    bootstrap_servers=["kafka:9092"],
    group_id="leaderboard-compute",
    value_deserializer=lambda m: json.loads(m.decode()),
    auto_offset_reset="earliest",
)

def process_batch(batch: dict[str, tuple[float, int]]):
    # batch maps player_id -> (highest_score_in_window, timestamp)
    if not batch:
        return
    pipe = r.pipeline(transaction=False)
    for player_id, (score, _ts) in batch.items():
        # GT flag handles monotonicity server-side as a safety net
        pipe.zadd(LEADERBOARD_KEY, {player_id: score}, gt=True)
    pipe.execute()

def run():
    batch: dict[str, tuple[float, int]] = {}
    window_start = time.time()

    for msg in consumer:
        event = msg.value
        player_id = event["player_id"]
        new_score = float(event["score"])
        ts = event["timestamp_ms"]

        # Within the batch window, keep only the highest score per player
        if player_id not in batch or new_score > batch[player_id][0]:
            batch[player_id] = (new_score, ts)

        # Flush batch every 200ms
        if time.time() - window_start >= BATCH_WINDOW_MS / 1000:
            process_batch(batch)
            batch = {}
            window_start = time.time()

At 50,000 updates per second with a 200ms batch window, each flush processes up to 10,000 updates. With deduplication per player in the batch window, the actual Redis ZADD calls are typically 8,000-9,000 per flush (players updating more than once per 200ms are merged). This keeps Redis write load at approximately 40,000-45,000 ops/sec - within one node’s capacity.

Rank Approximation for Personal Queries

When the leaderboard is served from a single Redis sorted set, personal rank is a trivial ZREVRANK call - O(log N). The problem appears when we shard the leaderboard across multiple Redis nodes for horizontal scaling. With 4 partitions, a player’s true global rank requires querying all 4 partitions and merging the results.

Partition-based rank approximation works as follows: each partition holds a contiguous score range (or a hash-based subset). To find a player’s global rank, we query their local partition rank and estimate their rank in the other partitions based on the score distribution.

# Partition-based rank approximation across 4 Redis shards
# Demonstrates: per-partition ZREVRANK + cross-partition count merge

import redis
from typing import Optional

# 4 Redis nodes, each holding ~2.5M players
# Partitioned by score range: [0,250), [250,500), [500,750), [750+)
SCORE_RANGES = [
    (0, 249),
    (250, 499),
    (500, 749),
    (750, float("inf")),
]
REDIS_NODES = [
    redis.Redis(host=f"leaderboard-redis-{i}", decode_responses=True)
    for i in range(4)
]
LEADERBOARD_KEY = "tournament:2026:leaderboard"

def get_partition(score: float) -> int:
    for i, (lo, hi) in enumerate(SCORE_RANGES):
        if lo <= score <= hi:
            return i
    return len(SCORE_RANGES) - 1

def get_approximate_global_rank(player_id: str, player_score: float) -> Optional[int]:
    partition_idx = get_partition(player_score)
    local_r = REDIS_NODES[partition_idx]

    # Step 1: get rank within own partition (exact)
    local_rank = local_r.zrevrank(LEADERBOARD_KEY, player_id)
    if local_rank is None:
        return None

    # Step 2: count members with higher scores in other partitions
    # For score-range partitions, all members in higher-score partitions outrank us
    higher_partition_count = 0
    for i, r_node in enumerate(REDIS_NODES):
        if i <= partition_idx:
            continue
        # All members in partition i have scores > our partition's max
        count = r_node.zcard(LEADERBOARD_KEY)
        higher_partition_count += count

    # Step 3: global rank = members above us in higher partitions + local rank + 1
    global_rank = higher_partition_count + local_rank + 1
    return global_rank

def get_exact_top_100() -> list[dict]:
    # For the top-N list, query only the highest-score partition first
    # If it has < 100 members, spill into the next partition
    results = []
    for i in range(len(REDIS_NODES) - 1, -1, -1):
        needed = 100 - len(results)
        if needed <= 0:
            break
        entries = REDIS_NODES[i].zrange(
            LEADERBOARD_KEY, 0, needed - 1, rev=True, withscores=True
        )
        results.extend(entries)
    return [
        {"rank": i + 1, "player_id": pid, "score": int(score)}
        for i, (pid, score) in enumerate(results[:100])
    ]

Data Model

The leaderboard data model separates three concerns: the live sorted set in Redis, the player profile store, and the event log for audit and replay.

-- PostgreSQL: player profile and tournament registration
-- Demonstrates: player metadata decoupled from live ranking store

CREATE TABLE players (
    player_id       TEXT        PRIMARY KEY,
    display_name    TEXT        NOT NULL,
    avatar_url      TEXT,
    region          TEXT        NOT NULL DEFAULT 'global',
    created_at      TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE TABLE tournament_registrations (
    tournament_id   TEXT        NOT NULL,
    player_id       TEXT        NOT NULL REFERENCES players(player_id),
    registered_at   TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    PRIMARY KEY (tournament_id, player_id)
);

-- Score event log: audit trail and replay source
CREATE TABLE score_events (
    id              BIGSERIAL   PRIMARY KEY,
    tournament_id   TEXT        NOT NULL,
    player_id       TEXT        NOT NULL,
    score           BIGINT      NOT NULL,
    previous_score  BIGINT      NOT NULL DEFAULT 0,
    source          TEXT        NOT NULL,  -- 'match_end', 'bonus', 'correction'
    applied         BOOLEAN     NOT NULL DEFAULT TRUE,
    event_ts        TIMESTAMPTZ NOT NULL,
    received_at     TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE INDEX ON score_events (tournament_id, player_id, event_ts DESC);
CREATE INDEX ON score_events (tournament_id, received_at DESC);

The Redis data structures form a companion layer to Postgres:

Redis key patterns:
  tournament:{id}:leaderboard          Sorted Set    player_id -> score, 10M members
  tournament:{id}:top100:cache         String (JSON) cached ZRANGE result, 1s TTL
  tournament:{id}:score:{player_id}    String        last-applied score, used for dedup
  tournament:{id}:meta                 Hash          start_time, end_time, player_count

Tie-breaking strategy: Redis sorted sets break score ties by lexicographic order of member names. To encode time-of-achievement as a tiebreaker, we embed a timestamp suffix in the score: actual_score * 10^6 + (MAX_TS - achievement_ts_ms). Higher scores rank higher; among ties, earlier achievement time (larger MAX_TS - ts) ranks higher. This avoids a separate tiebreaker field while keeping the sort key a single float64.

# Tie-breaking score encoding: embed timestamp for earlier-is-better tiebreak
# Demonstrates: packing score + time into a single float64 sort key

import time

MAX_TS_MS = 10_000_000_000  # 10^10, safely larger than any real tournament timestamp
SCORE_MULTIPLIER = 10**7    # supports scores up to 10M with ms-resolution tiebreak

def encode_score(raw_score: int, achievement_ts_ms: int) -> float:
    # Higher raw_score always wins; for equal scores, earlier achievement ranks higher
    tiebreak = MAX_TS_MS - achievement_ts_ms
    return float(raw_score * SCORE_MULTIPLIER + tiebreak)

def decode_score(encoded: float) -> tuple[int, int]:
    encoded_int = int(encoded)
    raw_score = encoded_int // SCORE_MULTIPLIER
    tiebreak = encoded_int % SCORE_MULTIPLIER
    achievement_ts_ms = MAX_TS_MS - tiebreak
    return raw_score, achievement_ts_ms

# Example:
# Player A scores 1000 at t=1000ms -> encode_score(1000, 1000) = 10_000_009_999_000_000
# Player B scores 1000 at t=2000ms -> encode_score(1000, 2000) = 10_000_009_998_000_000
# A > B in sort order, so A ranks higher (achieved the score first)

Key Algorithms and Protocols

Atomic Score Update with Lua

While ZADD GT handles the basic monotonicity case, a production system needs to also log the previous score and enforce tournament-specific business rules atomically. For cases requiring more than one Redis command to be atomic, we use a Lua script.

# Lua script: atomic score update with previous-score capture
# Demonstrates: Lua atomicity for multi-step score update in Redis

import redis

r = redis.Redis(host="leaderboard-redis", decode_responses=True)

ATOMIC_UPDATE_SCRIPT = """
-- KEYS[1] = leaderboard sorted set key
-- KEYS[2] = player's last-score string key
-- ARGV[1] = player_id
-- ARGV[2] = new encoded score (float string)
local current = redis.call('ZSCORE', KEYS[1], ARGV[1])
local new_score = tonumber(ARGV[2])
if current == false or tonumber(current) < new_score then
    redis.call('ZADD', KEYS[1], new_score, ARGV[1])
    redis.call('SET', KEYS[2], ARGV[2], 'EX', 86400)
    return {1, current or '0'}   -- {updated=true, previous_score}
end
return {0, current}              -- {updated=false, current_score}
"""

atomic_update = r.register_script(ATOMIC_UPDATE_SCRIPT)

def update_score_atomic(
    tournament_id: str, player_id: str, encoded_score: float
) -> tuple[bool, float]:
    lb_key = f"tournament:{tournament_id}:leaderboard"
    score_key = f"tournament:{tournament_id}:score:{player_id}"
    result = atomic_update(
        keys=[lb_key, score_key],
        args=[player_id, str(encoded_score)],
    )
    updated = bool(result[0])
    previous = float(result[1])
    return updated, previous

Top-N Cache Invalidation Strategy

Serving 200,000 top-100 reads per second from Redis directly would require approximately 200K ZRANGE calls per second - around 2x Redis’s comfortable throughput for a 100-element range scan. Instead, we cache the serialized top-100 list behind a 1-second TTL.

The critical question is: how do we invalidate the cache after a top-100 member’s score changes, without re-fetching the full list on every score update?

# Top-N cache with smart invalidation: only refresh when top-100 changes
# Demonstrates: sentinel key trick to detect top-100 membership changes

import redis
import json
import time

r = redis.Redis(host="leaderboard-redis", decode_responses=True)

LEADERBOARD_KEY = "tournament:2026:leaderboard"
TOP100_CACHE_KEY = "tournament:2026:top100:cache"
TOP100_MEMBERS_KEY = "tournament:2026:top100:members"  # set of current top-100 IDs
TOP100_TTL = 1  # 1 second TTL for top-100 cache

def refresh_top100_cache() -> list[dict]:
    entries = r.zrange(LEADERBOARD_KEY, 0, 99, rev=True, withscores=True)
    result = [
        {"rank": i + 1, "player_id": pid, "score": int(score)}
        for i, (pid, score) in enumerate(entries)
    ]
    # Cache the serialized result
    r.setex(TOP100_CACHE_KEY, TOP100_TTL, json.dumps(result))
    # Track top-100 member IDs for fast membership checking
    pipe = r.pipeline()
    pipe.delete(TOP100_MEMBERS_KEY)
    if result:
        pipe.sadd(TOP100_MEMBERS_KEY, *[e["player_id"] for e in result])
    pipe.expire(TOP100_MEMBERS_KEY, 10)
    pipe.execute()
    return result

def get_top100() -> list[dict]:
    cached = r.get(TOP100_CACHE_KEY)
    if cached:
        return json.loads(cached)
    return refresh_top100_cache()

def should_invalidate_top100(player_id: str) -> bool:
    # Only invalidate the cache if the updated player is IN the top 100
    # or if their new score is high enough to enter the top 100
    is_member = r.sismember(TOP100_MEMBERS_KEY, player_id)
    if is_member:
        return True
    # Check if this player's new score would enter the top 100
    bottom_of_top100 = r.zrange(LEADERBOARD_KEY, 99, 99, rev=True, withscores=True)
    return False  # simplified: a background refresher handles TTL-based refresh

In practice, the top-100 cache is refreshed by a dedicated background coroutine every 500ms rather than on every write. The 1-second TTL ensures staleness is bounded even if the refresher falls behind.

Scaling and Performance

The capacity math is straightforward, and it reveals why a single Redis node is viable during peak load:

Capacity Estimation:

Storage:
  10M players * 40 bytes/entry = 400MB per sorted set
  Hash table index overhead (1.5x) = 200MB additional
  Total Redis memory: ~600MB for leaderboard + 200MB for auxiliary keys = ~800MB
  Single Redis node with 4GB RAM: comfortable headroom

Write throughput:
  50,000 score updates/sec at peak
  Batch window 200ms: up to 10,000 updates per flush
  Deduplication within window: ~9,000 unique ZADD calls per flush
  ZADD throughput per Redis node: ~100,000-150,000 ops/sec
  Headroom: 1 node handles peak load with ~50% spare capacity

Read throughput:
  200,000 top-100 reads/sec -> served by CDN/cache, not Redis
  Cache hit rate target: 99.5% (1s TTL, CDN)
  Redis ZRANGE calls: 200,000 * 0.005 = 1,000/sec (cache misses only)
  ZREVRANK calls (personal rank): up to 50,000/sec
  Combined Redis read load: ~51,000 ops/sec (well within 100K/sec limit)

Kafka:
  50,000 events/sec * 200 bytes/event = 10 MB/sec ingress
  Partitioned by player_id, 30 partitions, 8-hour tournament
  Retention: 10 MB/sec * 8 * 3600 = ~288 GB (manageable)

When a single sorted set exceeds comfortable single-node capacity - either in memory or ops/sec - we move to a partitioned leaderboard design:

• Phase 1 (up to 10M players, single node): one global sorted set, 600MB, single Redis node
• Phase 2 (10M-50M players, 4 partitions): score-range sharding into 4 sorted sets, each on its own node
• Phase 3 (50M+ players): each partition shard gets 2 read replicas; writes go to primary, top-100 reads fan out to replicas

Failure Modes and Recovery

Failure	Detection	Impact	Recovery
Redis primary node crash	Redis Sentinel health check alert, PING timeout	Score updates queue in Kafka; top-100 cache serves stale data	Sentinel promotes replica in 10-30 seconds; Kafka consumer resumes; stale cache acceptable during failover
Kafka lag spike (consumer falling behind)	Consumer group lag > 50K events	Score updates delayed; leaderboard freshness degrades past 1s SLO	Scale up Leaderboard Compute Service instances; Kafka absorbs backlog automatically
Score-range partition skew	Prometheus ZCARD per partition; skew > 3x	Hot partition overloads its Redis node	Rebalance score ranges; migrate members across partitions with ZUNIONSTORE
Score replay during reconnect	Duplicate event_id in score_events log	Score counted twice if ZADD GT guard is bypassed	ZADD GT is idempotent for equal scores; duplicate events with equal scores are no-ops
Top-N cache miss storm	CDN cache eviction + Redis ZRANGE spike	ZRANGE throughput saturates Redis on cache cold-start	Pre-warm cache before tournament start; rate-limit cache miss requests with a mutex pattern (one refresh in-flight)
Leaderboard data loss (Redis wipe)	DBSIZE drops to 0; Kafka consumer sees no matching keys	Leaderboard reset to empty	Replay from Kafka score_events topic and score_events Postgres table; rebuild sorted set in minutes

Comparison of Approaches

Approach	Write Throughput	Top-100 Latency	Exact Accuracy	Memory Cost	Best Fit
Redis sorted set (this design)	50K/sec per node	Under 1ms (cached)	Yes (top-100 exact)	~600MB for 10M	Tournament-scale, memory-bound
PostgreSQL ORDER BY + rank window	~5K writes/sec	50-500ms (no index)	Yes	~2GB (indexed)	Small leaderboards, strong consistency
PostgreSQL + materialized rank column	~10K writes/sec	5-10ms	Yes (stale)	~1.5GB	Medium scale, infrequent rank queries
Distributed counter sharding	500K/sec	N/A (no rank)	No (count only)	Low	View counts, not ranked lists
HyperLogLog approximation	Very high	Under 1ms	No (approximate)	12KB	Unique player counts, not ranking
Cassandra sorted by score	100K/sec	10-50ms	Yes	~1.2GB	Write-heavy, approximate rank ok

The Redis sorted set wins for tournament leaderboards because it provides exact ranking in O(log N) with the entire dataset in memory. The competition from Postgres materializes only when the dataset exceeds Redis memory limits (hundreds of millions of players) or when strong transactional consistency is required (payment-linked rankings). For a time-boxed tournament with 10 million players, Redis is unambiguously the right tool.

Key Takeaways

• Redis sorted sets are built for this problem: ZADD GT + ZREVRANK + ZRANGE together cover ingest, personal rank, and top-N queries with O(log N) guarantees - no secondary index, no full scan, no aggregation job.
• Monotonicity is free with ZADD GT: the GT flag makes score-only-increases a single atomic operation, avoiding a read-then-write race that would require distributed locking.
• Batching is the write throughput multiplier: collapsing 200ms of score updates per player into one ZADD reduces Redis write ops by 10-50x during match-end burst windows.
• Score-range partitioning makes approximation exact: by assigning contiguous score ranges to partitions, cross-partition rank estimation requires only a ZCARD per foreign partition, not a score distribution sample.
• The top-100 cache is the read path’s primary scaling mechanism: 200K reads/sec served from CDN with a 1-second TTL means Redis handles only ~1K cache misses per second, not 200K ZRANGE calls.
• Tie-breaking belongs in the score encoding: packing timestamp into the float64 score value keeps the sort key a single field, avoids secondary sorting passes, and works with Redis’s native lexicographic tiebreak.
• Single Redis node handles 10M players: the dataset fits in 600MB and write load (50K/sec) is within single-node throughput, making the simpler single-node design preferable to premature sharding.
• Cache miss storms need probabilistic early refresh: the thundering herd on TTL expiry is more dangerous than cache staleness; refresh before expiry with low probability per request rather than reacting to TTL expiry.

The biggest mental shift in designing this system is accepting that rank is not stored anywhere - it is computed on demand from the sorted structure in O(log N). You never write a rank column. You never run a ranking query. The rank is a property of the sorted set’s current state, materialized only when queried. This inversion - from storing rank to querying for position - is what allows 10 million concurrent ranked players to be served from a single in-memory data structure.

Frequently Asked Questions

Q: Why not just use ORDER BY in Postgres?

A: ORDER BY score DESC with a LIMIT 100 on an indexed column runs in roughly O(100) time if the index covers the query - fast for cold reads. The problem is the personal rank query: SELECT COUNT(*) FROM leaderboard WHERE score > $1 requires scanning the entire index for every rank query. At 50K score updates per second and 50K personal rank queries per second, you would need to maintain a materialized rank column that is updated on every score change - and updating a rank column for one player’s score change could cascade to updating millions of rows (everyone below them shifts down by one). Redis sorted sets avoid this entirely: rank is computed structurally, not stored.

Q: How do you handle ties?

A: We encode the achievement timestamp into the score float64 using encoded = raw_score * 10^7 + (MAX_TS - achievement_ts_ms). Players with the same raw score are differentiated by when they achieved that score, with earlier being better. This produces a total ordering with no ties in the sorted set, so ZREVRANK always returns a unique rank. The display layer can decode the raw score and timestamp separately to show “tied at 1000 points” while maintaining a definitive rank ordering.

Q: What if a score update arrives out of order?

A: Because we use ZADD GT (only update if new score is greater), out-of-order delivery of a lower score after a higher one is silently dropped. The GT flag makes score updates idempotent with respect to ordering: applying [100, 200, 150] in order results in the same state as applying [200]. The Kafka player_id-partitioned topic ensures per-player ordering, so out-of-order delivery only occurs when Kafka consumers retry after a failure - and in that case, the GT guard protects correctness.

Q: How would this design scale to 100 million players?

A: At 100 million players, the single sorted set requires 6GB of memory - feasible on a large Redis instance but with less headroom. More critically, at 5x the player count, score update throughput likely scales proportionally, potentially reaching 250K ops/sec. At that point, we move to 8-10 score-range partitions on separate Redis nodes. The personal rank approximation becomes multi-hop: query your partition for local rank, sum ZCARD of all higher-score partitions, sum. The top-100 query only ever touches the highest-score partition unless it has fewer than 100 members, so read latency is unaffected by sharding.

Q: What happens if the Redis node running the leaderboard crashes during the tournament?

A: Redis Sentinel detects the failure and promotes a replica in 10-30 seconds. During that window, the Leaderboard Compute Service stops processing Kafka events (it cannot write to Redis) and the top-100 cache serves stale data from CDN. Score update events accumulate in Kafka (with 24-hour retention). After the replica promotion, the Compute Service resumes consuming from Kafka and applies all missed updates. The leaderboard catches up to real-time within minutes of recovery. Players see stale rankings for up to 30 seconds during failover - a known and documented SLO exception for failover events.

Q: Why use Kafka between game servers and the Leaderboard Compute Service instead of writing directly to Redis?

A: Two reasons. First, Kafka absorbs match-end burst traffic: when thousands of matches end simultaneously in the final minute of a tournament round, score submissions spike 4-8x above average. Redis can handle the average load easily, but the burst would saturate it without a buffer. Kafka gives us elastic absorption. Second, Kafka provides a replay log: if the leaderboard state needs to be reconstructed (Redis data loss, bug causing incorrect scores), we can replay the Kafka topic and rebuild the sorted set deterministically from the event history.

Interview Questions

Q: How would you design the leaderboard if scores could decrease as well as increase (for example, a time-penalty system)?

Expected depth: The ZADD GT trick no longer works because a score decrease must also be applied. You need a compare-and-swap: read the current score, verify it matches what you expect, then write the new score. In Redis this requires a Lua script or a WATCH/MULTI/EXEC transaction. Discuss throughput impact of Lua vs transactions. Note that out-of-order delivery becomes more dangerous - a delayed decrease applied after a subsequent increase would incorrectly lower the score. You need sequence numbers or vector clocks per player, not just monotonicity checks.

Q: The top-100 leaderboard is served to a broadcast overlay at 60fps. Each frame must show the latest rankings. How do you serve this without hammering Redis at 60 requests per second per overlay instance?

Expected depth: Server-sent events or WebSocket push. The Leaderboard Compute Service publishes top-100 diffs (not full snapshots) to a pub/sub channel (Redis Pub/Sub or a dedicated WebSocket server) after each successful batch write. Broadcast overlays subscribe to the channel and apply incremental updates. Discuss why diffing (only sending changed entries) is important when 99 of the 100 top entries are stable and only 1 or 2 change per second. Mention that this requires a stateful connection which complicates horizontal scaling of the broadcast server.

Q: How would you detect and prevent leaderboard cheating where a game client submits fabricated high scores?

Expected depth: Score validation is a game server responsibility, not a leaderboard responsibility - the leaderboard trusts its input. But we can add anomaly detection: flag players whose score velocity exceeds the game engine’s maximum possible rate (if you can score at most 50 points per minute, a 10,000-point jump in 1 second is impossible). Discuss signing score events with a game server secret key so the leaderboard service can reject unsigned updates. Mention that this is defense in depth - the primary protection is server-authoritative game logic that never trusts client-submitted scores.

Q: How would you roll out a bug fix that discovers 500 players have incorrect scores and need to be corrected downward?

Expected depth: Downward corrections break the ZADD GT invariant. You would use a Lua script that does an unconditional ZADD (without GT) for the affected players, then logs the correction to the score_events table with source='correction'. Discuss the operational risk of doing this during a live tournament versus waiting for a dead period. Note that downward corrections shift every other player’s rank upward, which could trigger cached top-100 invalidations for many CDN edges. Propose running the correction as a Lua pipeline during a low-traffic period and pre-warming the top-100 cache immediately after.

Q: Walk through the memory and throughput math for your design at 50 million players.

Expected depth: 50M * 40 bytes = 2GB for sorted set + ~1GB for hash table index = ~3GB. A 32GB Redis node handles this comfortably. At 5x player count, expect ~250K score updates/sec at peak. Single Redis node throughput is ~100K-150K ops/sec for sorted set operations, so 250K exceeds single-node capacity. Move to 4 score-range partitions, each handling ~62.5K ops/sec. Discuss that the top-100 cache only ever touches the highest-score partition and CDN absorbs 99%+ of reads, so read throughput is not the scaling constraint - write throughput is.