Picture an old telethon studio: a wall of phone lines, one clicker counter bolted next to each phone, and an operator whose whole job is to walk the wall every few seconds, add up all the clickers, and shout the running total to the host standing in front of the camera. No single clicker knows the total. No single phone line has to wait for any other phone line to be answered before its own call counts. The operator’s job isn’t to answer every call personally, it’s to sum numbers that are already being counted in parallel, on a schedule fast enough that the number on air never looks stale.

That is the shape of a live poll during a broadcast. A host asks a question, puts a QR code or a button on screen, and somewhere between a few hundred thousand and a million viewers tap an option within a thirty-second window, no login screen, no account creation, nothing standing between the tap and the vote. The number displayed back to the audience has to keep climbing in near real time, because the climbing number is part of the entertainment, and it has to be trustworthy enough that nobody can look at the result and reasonably claim one option won because a script hammered the vote button.

The naive version of this system is a single row in a database: UPDATE poll_options SET vote_count = vote_count + 1 WHERE option_id = ?. It works perfectly in a demo. It falls over the instant real load hits it, because every one of a million concurrent voters is now queued behind a single row lock, and the queue itself becomes the bottleneck long before the database’s raw throughput does. The opposite naive version, batch every vote and only compute the tally once the poll closes, avoids the lock contention entirely but throws away the one thing that makes a live poll compelling: a number the audience watches move in front of their eyes. Both mistakes come from treating “count the votes” and “show the count” as the same problem solved the same way, when at this scale they need genuinely different mechanisms, one optimized for absorbing a burst of writes, the other for cheaply reading a number that is allowed to be slightly stale.

We need to solve for three things simultaneously: an ingestion and counting path that can absorb a million votes in a thirty-second window without any single counter becoming a point of contention, a way to tell one honest vote from a script replaying the same request thousands of times a second, without ever asking anyone to sign in, and a broadcast path that keeps a live number moving on millions of screens without turning every single vote into a message sent to every single viewer.

Requirements and Constraints

Functional Requirements

• Accept a vote submission tied to a specific poll option during a bounded voting window, without requiring the voter to create an account or log in
• Treat every vote submission as idempotent: retries of the same client request, caused by a flaky network or a double-tap, must never increment the tally more than once
• Deduplicate votes from the same voter across multiple submissions using non-account signals (device fingerprint, session token, IP plus client entropy), so at most one vote per voter counts toward the tally at any moment, with vote changes allowed before the poll closes (last vote wins)
• Maintain a continuously updating tally per poll option, visible to every connected viewer, not just the ones who voted
• Broadcast tally updates to every connected viewer over a push channel (WebSocket) rather than requiring viewers to poll an HTTP endpoint for updates
• Score every vote against anti-stuffing signals (submission velocity, device fingerprint reuse, IP reputation, connection age, behavioral timing) and down-weight or shadow-filter suspicious votes without immediately telling the sender they were caught
• Close the poll at a fixed time and publish a final, exact, reconciled tally that resolves any approximate counting drift accumulated during the live window

Non-Functional Requirements

• Scale: 1,000,000 concurrent votes during a 30-second live voting window, sustained around 33,000 votes/sec and bursting above 100,000 votes/sec at the open and close of the window; up to 5,000,000 concurrent viewer WebSocket connections watching the tally update
• Latency: vote submission acknowledgment at p99 under 150ms; a tally update must reach a connected viewer within a bounded consistency window of 2 seconds of the underlying vote landing
• Durability: every accepted vote is durably recorded before its client receives a success acknowledgment, so no vote is lost even if a shard counter process crashes moments later
• Correctness guarantee: the final, closed-poll tally must be exact, not approximate, and must equal the true count of deduplicated, trust-cleared votes
• Availability: vote ingestion targets 99.95% uptime during the live broadcast window, since a voting endpoint that is down during a live TV moment cannot be meaningfully recovered after the fact
• Abuse resistance: sustain at least a 50x single-source vote-stuffing attempt (one device or script submitting votes far above any human-plausible rate) without materially moving the published tally

Constraints and Assumptions

• Viewer authentication, account creation, and login are explicitly out of scope; voter identity for deduplication purposes is inferred from device and session signals, not a durable account system
• The video and broadcast delivery pipeline, the live stream itself, is a separate system; this design owns only the poll widget’s vote submission, tally computation, and result broadcast
• We assume a CDN and edge network already exists for static assets and can terminate the bulk of inbound WebSocket connections close to viewers
• Cross-poll analytics, historical poll browsing, and long-term archival beyond the single broadcast are out of scope
• We assume one poll is open at a time per broadcast, though the design generalizes cleanly to many concurrent independent polls, one per live channel

High-Level Architecture

The system has five major components: the Vote Ingestion Gateway, the Voter Identity and Anti-Stuffing Engine, the Sharded Vote Counters, the Tally Aggregator, and the WebSocket Broadcast Fan-out.

The Vote Ingestion Gateway is the front door: it validates a vote’s shape, enforces the idempotency key so retries never double count, and applies coarse per-device rate limiting before anything expensive happens. The Voter Identity and Anti-Stuffing Engine resolves who is voting without ever asking for a login, deduplicates repeat submissions from the same voter, and scores every vote for how likely it is to be scripted abuse rather than a real tap. The Sharded Vote Counters are the hot write path, a bank of independent atomic counters per option that absorb the actual increment volume without any writer contending with any other writer. The Tally Aggregator is the operator walking the wall of clickers, it sums the shard counters on a fixed cadence and produces a single approximate-but-fresh snapshot of the current standings. The WebSocket Broadcast Fan-out takes that snapshot and pushes it out to every connected viewer as a small diff, not a full resend, spread across many fan-out pods so no single process has to hold millions of connections.

A vote’s life looks like this: a viewer taps an option, the client attaches a locally generated idempotency key and whatever device signals are available, and the request lands at the gateway. The gateway checks the idempotency key, the identity engine resolves the voter’s token and scores the vote’s trustworthiness, and if the vote clears the trust floor it gets hashed to one of several shard counters for that option and incremented atomically. Independently, on a fixed interval, the aggregator sums every shard across every option and publishes a snapshot; the fan-out tier diffs that snapshot against what it last sent and pushes only the changed numbers to every viewer it holds a connection for. When the poll’s close time arrives, the system stops treating the shard counters as authoritative and instead replays the durable vote ledger to compute an exact, reconciled final count.

Component Deep Dives

The Vote Ingestion Gateway

This component’s job is to turn a raw HTTP or WebSocket vote submission into a validated, idempotent request before any counting logic ever runs.

The obvious mistake is checking idempotency after doing the expensive work, resolving identity, scoring trust, incrementing a shard, and only then noticing the request was a duplicate. Under a retry storm, that ordering means every duplicate still pays the full cost of processing, which is exactly the load pattern most likely to appear during the first and last seconds of a voting window when clients are retrying against a suddenly-busy endpoint. The gateway instead makes the idempotency check the very first gate a request passes through, using a fast SET NX against a shared cache so a duplicate is recognized and returned the exact same acknowledgment it got the first time, without touching anything downstream.

# Idempotent vote intake: SET NX with the client's idempotency key guarantees
# a retried request is recognized and acknowledged without re-processing
import time
import redis

r = redis.Redis(host="idempotency-cache", decode_responses=True)
IDEMPOTENCY_TTL_SECONDS = 60  # covers realistic client retry windows

def accept_vote(poll_id: str, option_id: str, idempotency_key: str,
                 voter_token: str, raw_signals: dict) -> dict:
    dedupe_key = f"idem:{poll_id}:{idempotency_key}"
    claimed = r.set(dedupe_key, "1", nx=True, ex=IDEMPOTENCY_TTL_SECONDS)
    if not claimed:
        # Same request already accepted once; return the same ack, do not requeue.
        return {"status": "already_accepted", "idempotency_key": idempotency_key}

    envelope = {
        "poll_id": poll_id,
        "option_id": option_id,
        "voter_token": voter_token,
        "idempotency_key": idempotency_key,
        "signals": raw_signals,
        "received_at": time.time(),
    }
    enqueue_for_dedup(envelope)
    return {"status": "accepted", "idempotency_key": idempotency_key}

def enqueue_for_dedup(envelope: dict) -> None:
    import json
    r.xadd(f"votes:{envelope['poll_id']}:stream",
           {"payload": json.dumps(envelope, separators=(",", ":"))})

This is the same trick a payment gateway uses to make sure double-tapping “Pay Now” never charges a card twice: the client, not the server, mints a unique request id up front, and the server’s very first job is to check whether it has ever seen that id before doing anything else. What breaks without it: a viewer on a shaky mobile connection who taps once but whose client retries three times over a slow network ends up counted three times, and at a million-voter scale that pattern alone can shift a close result.

The Voter Identity and Anti-Stuffing Engine

This component’s job is to decide, without an account system to consult, whether a vote came from a distinct, plausible human, and to attach a continuous trust score to every vote rather than a binary allow or block.

A smart engineer’s first instinct is to block anything that looks automated outright, reject the request with an error the instant a signal looks suspicious. That instinct is wrong for two reasons: a hard block trains an attacker to probe for the exact threshold and adjust until they slip under it, and a false positive on a genuinely fast, enthusiastic human voter turns into a visible, undeserved rejection. Instead every vote gets a continuous trust score between zero and one, computed from several independent signals, and only votes below a floor are excluded from the tally, silently, without telling the sender anything changed.

# Vote identity resolution + trust scoring: a repeat voter_token updates
# their existing vote (last-vote-wins) instead of creating a second counted
# vote, and every vote gets a continuous trust score used to filter abuse
TRUST_FLOOR = 0.15  # below this, a vote is shadow-counted, not dropped

def trust_score(signals: dict) -> float:
    score = 1.0

    # Requests arriving faster than a human can plausibly tap twice.
    interval = signals.get("seconds_since_last_vote_this_device")
    if interval is not None and interval < 0.4:
        score *= 0.2

    # A device/IP pair issuing far more votes than one human plausibly casts.
    votes_from_ip_last_minute = signals.get("votes_from_ip_last_minute", 1)
    if votes_from_ip_last_minute > 20:
        score *= max(0.05, 20 / votes_from_ip_last_minute)

    # A WebSocket connection opened only seconds before voting looks scripted;
    # a viewer who has been watching for minutes looks organic.
    conn_age = signals.get("ws_connection_age_seconds", 999)
    if conn_age < 2.0:
        score *= 0.5

    # Known-bad ASN / datacenter IP ranges (proxies, cloud scripts) rather
    # than residential or mobile carrier ranges.
    if signals.get("asn_is_datacenter", False):
        score *= 0.3

    return max(0.0, min(1.0, score))

def resolve_vote(existing_vote: dict | None, new_option_id: str,
                  signals: dict, now: float) -> dict:
    score = trust_score(signals)
    action = "insert" if existing_vote is None else "change"
    if existing_vote and existing_vote["option_id"] == new_option_id:
        action = "noop"  # re-submission of the same choice, not a real change
    return {
        "action": action,
        "previous_option_id": existing_vote["option_id"] if existing_vote else None,
        "new_option_id": new_option_id,
        "trust_score": score,
        "counted": score >= TRUST_FLOOR,
        "resolved_at": now,
    }

Think of this like a polling place clerk checking a name off a printed roll before handing over a ballot, except there is no printed roll, so the clerk instead infers “have I seen this exact combination of device signals before” from a fingerprint rather than a name. What breaks without last-vote-wins semantics: a viewer who changes their mind partway through the window, taps a different option, either gets rejected outright or ends up double counted across two options, both of which are worse outcomes than simply moving their one vote.

The Sharded Vote Counters

This component’s job is to absorb the actual increment volume, up to and beyond 100,000 writes per second during a burst, without any two writers ever contending for the same counter.

A single counter row per option, even one backed by an in-memory store instead of a locked database row, still serializes every writer that touches it. At 100,000 writes/sec against one key, even a store capable of millions of ops/sec per key starts to show queueing latency once network round-trips and connection multiplexing enter the picture, and that queueing is exactly what a p99 submission latency budget of 150ms cannot absorb. The fix is to split each option’s count into a fixed number of independent shard counters and hash each voter deterministically to exactly one shard, so writes for the same option land on different physical keys and never wait on each other.

# Sharded increment: a vote is hashed to one of N shards per option so
# concurrent writers never contend on the same counter key
import hashlib

SHARDS_PER_OPTION = 16

def shard_for_voter(voter_token: str, shards: int = SHARDS_PER_OPTION) -> int:
    digest = hashlib.blake2b(voter_token.encode(), digest_size=8).digest()
    return int.from_bytes(digest, "big") % shards

def apply_counted_vote(redis_client, poll_id: str, resolved: dict, voter_token: str) -> None:
    shard = shard_for_voter(voter_token)
    if resolved["action"] in ("insert", "change") and resolved["counted"]:
        if resolved["previous_option_id"]:
            prev_key = f"cnt:{poll_id}:{resolved['previous_option_id']}:{shard}"
            redis_client.decr(prev_key)
        new_key = f"cnt:{poll_id}:{resolved['new_option_id']}:{shard}"
        redis_client.incr(new_key)

This is the same technique a county elections office uses when it lets every precinct count its own ballot box independently rather than funneling every physical ballot through one central counting table. What breaks without sharding: every vote for a popular option contends for the exact same key, and the shard count effectively drops to one, reproducing the same lock-contention bottleneck a single global counter has, just implemented in a faster store.

The Tally Aggregator

This component’s job is to sum every shard counter for every option on a fixed cadence and publish a single, coherent snapshot, rather than letting every individual increment ripple out as its own message.

Publishing a broadcast message for every single vote sounds maximally live, but it means write volume and broadcast volume are the same number, and at 100,000 votes/sec that is 100,000 messages/sec that somehow have to reach 5,000,000 viewer connections, an amplification that no fan-out tier can sustain. The aggregator decouples the two by reading all shard counters on its own schedule and producing one snapshot per interval regardless of how many votes landed during that interval, turning an unbounded, bursty write rate into a fixed, predictable publish rate.

// Tally aggregator: sums N shard counters per option on a fixed cadence
// and publishes a snapshot rather than one message per vote
package tally

import (
	"context"
	"strconv"
	"time"

	"github.com/redis/go-redis/v9"
)

const shardsPerOption = 16
const publishInterval = 500 * time.Millisecond

func aggregateOnce(ctx context.Context, rdb *redis.Client, pollID string, optionIDs []string) (map[string]int64, error) {
	totals := make(map[string]int64, len(optionIDs))
	for _, opt := range optionIDs {
		keys := make([]string, shardsPerOption)
		for s := 0; s < shardsPerOption; s++ {
			keys[s] = "cnt:" + pollID + ":" + opt + ":" + strconv.Itoa(s)
		}
		vals, err := rdb.MGet(ctx, keys...).Result()
		if err != nil {
			return nil, err
		}
		var sum int64
		for _, v := range vals {
			if v == nil {
				continue
			}
			n, _ := strconv.ParseInt(v.(string), 10, 64)
			sum += n
		}
		totals[opt] = sum
	}
	return totals, nil
}

func RunAggregatorLoop(ctx context.Context, rdb *redis.Client, pollID string, optionIDs []string, publish func(map[string]int64)) {
	ticker := time.NewTicker(publishInterval)
	defer ticker.Stop()
	for {
		select {
		case <-ctx.Done():
			return
		case <-ticker.C:
			totals, err := aggregateOnce(ctx, rdb, pollID, optionIDs)
			if err == nil {
				publish(totals)
			}
		}
	}
}

Think of this like the telethon operator’s walk down the wall of clickers: the operator doesn’t shout a new total after every single call, they walk the wall on their own rhythm and shout whatever the sum happens to be at that moment. What breaks without a fixed publish cadence: the aggregation cost couples directly to vote volume instead of shard count, so the exact moment the system is under the most write pressure is also the moment it tries to publish the most often, compounding load right when the system can least afford it.

The WebSocket Broadcast Fan-out

This component’s job is to take the aggregator’s periodic snapshot and push it to every connected viewer as a small diff, spread across enough independent pods that no single process has to hold millions of sockets.

The naive approach opens one WebSocket server process and lets every viewer connect to it. That process runs out of file descriptors and memory long before it reaches a million connections, and even if it didn’t, a single process crashing would silently disconnect the entire audience at once. Each fan-out pod instead holds a bounded slice of the total viewer connections, subscribes to the same tally snapshots as every other pod, and computes its own diff against the last snapshot it sent, so a viewer only ever receives the numbers that actually changed.

// WebSocket fan-out: each pod holds a shard of viewer connections and
// re-publishes every aggregator snapshot as a small diff, not a full resend
package fanout

import (
	"encoding/json"
	"sync"
)

type Snapshot struct {
	PollID string           `json:"poll_id"`
	Totals map[string]int64 `json:"totals"`
	AsOf   int64            `json:"as_of_ms"`
}

type Pod struct {
	mu    sync.RWMutex
	conns map[string]chan []byte // viewer_id -> outbound channel
	last  Snapshot
}

func (p *Pod) OnSnapshot(next Snapshot) {
	p.mu.Lock()
	diff := diffTotals(p.last.Totals, next.Totals)
	p.last = next
	p.mu.Unlock()

	if len(diff) == 0 {
		return
	}
	payload, _ := json.Marshal(struct {
		PollID string           `json:"poll_id"`
		Diff   map[string]int64 `json:"diff"`
		AsOf   int64            `json:"as_of_ms"`
	}{next.PollID, diff, next.AsOf})

	p.mu.RLock()
	defer p.mu.RUnlock()
	for _, ch := range p.conns {
		select {
		case ch <- payload:
		default:
			// Slow consumer: drop this tick rather than block the fan-out loop;
			// the next snapshot supersedes it anyway.
		}
	}
}

func diffTotals(prev, next map[string]int64) map[string]int64 {
	diff := make(map[string]int64)
	for k, v := range next {
		if prev[k] != v {
			diff[k] = v
		}
	}
	return diff
}

The analogy is a radio station’s regional relay towers, each covering a slice of the listening area, all retransmitting the exact same broadcast rather than the studio trying to run a dedicated wire to every single radio in the city. What breaks without horizontal fan-out pods and diffing: connection count and message size both grow unbounded together, and a viewer base that spikes from a hundred thousand to five million mid-broadcast, which is exactly when a poll is at its most interesting, takes the entire broadcast layer down with it.

Data Model

The data model spans four entities: polls, poll options, an append-only vote ledger that is the system’s real source of truth, and periodic shard checkpoints used for crash recovery.

-- Polls: the bounded voting window a broadcast opens and closes
CREATE TABLE polls (
    poll_id        UUID        PRIMARY KEY,
    broadcast_id   TEXT        NOT NULL,
    question       TEXT        NOT NULL,
    opens_at       TIMESTAMPTZ NOT NULL,
    closes_at      TIMESTAMPTZ NOT NULL,
    status         TEXT        NOT NULL DEFAULT 'scheduled'
        CHECK (status IN ('scheduled', 'open', 'closed', 'reconciled')),
    CHECK (closes_at > opens_at)
);

-- Poll options: the fixed set of choices a vote can target
CREATE TABLE poll_options (
    option_id      UUID        PRIMARY KEY,
    poll_id        UUID        NOT NULL REFERENCES polls(poll_id),
    label          TEXT        NOT NULL,
    display_order  INT         NOT NULL
);

-- Vote ledger: append-only source of truth. A voter_token can appear more
-- than once (vote changes); the reconciliation query keeps only the latest.
CREATE TABLE vote_ledger (
    vote_id           BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    poll_id           UUID        NOT NULL REFERENCES polls(poll_id),
    option_id         UUID        NOT NULL REFERENCES poll_options(option_id),
    voter_token       TEXT        NOT NULL,
    idempotency_key   TEXT        NOT NULL,
    trust_score       REAL        NOT NULL CHECK (trust_score >= 0 AND trust_score <= 1),
    submitted_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    UNIQUE (poll_id, idempotency_key)
);

CREATE INDEX ON vote_ledger (poll_id, voter_token, submitted_at DESC);

-- Shard checkpoints: durable, periodic snapshots of the in-memory shard
-- counters, used to recover approximate live totals after a crash
CREATE TABLE shard_checkpoints (
    poll_id          UUID        NOT NULL REFERENCES polls(poll_id),
    option_id        UUID        NOT NULL REFERENCES poll_options(option_id),
    shard_id         INT         NOT NULL,
    vote_count       BIGINT      NOT NULL,
    checkpointed_at  TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    PRIMARY KEY (poll_id, option_id, shard_id)
);

vote_ledger is the only table treated as authoritative; every other structure in the system, the shard counters, the aggregator snapshots, the broadcast diffs, is a derived, disposable view over it. The UNIQUE (poll_id, idempotency_key) constraint is the durable second line of defense behind the gateway’s Redis-based idempotency check: if the cache evicted a key under memory pressure and let a duplicate slip through, the ledger insert itself fails on conflict and the duplicate is caught before it can be double-counted. The index on (poll_id, voter_token, submitted_at DESC) exists specifically to make the reconciliation query, which needs each voter’s latest choice, fast even against tens of millions of ledger rows.

A vote is born the moment a viewer taps an option: it clears the gateway’s idempotency check, gets resolved and trust-scored by the identity engine, and, if it clears the trust floor, is hashed into a shard counter and written as a new row in vote_ledger. That row never changes after it is written, a vote change is a brand-new row with a later submitted_at, not an update to the old one. Throughout the open window, the tally shown to viewers comes entirely from the shard counters and the aggregator’s periodic sums. The moment closes_at is reached, the system stops trusting the shard counters as the final word and instead runs the reconciliation query against the ledger, which becomes the number that ships as the poll’s official result.

Key Algorithms and Protocols

Idempotent Vote Submission

Covered in the Vote Ingestion Gateway section above, this is a SET NX check against a shared cache, O(1) per request, backed by a unique constraint on the durable ledger as a second, slower line of defense.

Trust Scoring

Covered in the Voter Identity and Anti-Stuffing Engine section above, the trust function combines several independent signals multiplicatively, each in O(1), into a continuous score in [0, 1] rather than a binary decision.

Counter Sharding and Merge

Covered in the Sharded Vote Counters and Tally Aggregator sections above, writes hash a voter token to one of N shards in O(1), and the aggregator sums across all shards for all options in O(shards * options) per publish tick, independent of vote volume.

Exact Reconciliation via Ledger Replay

At poll close, the shard counters stop being authoritative and the system recomputes the tally directly from the durable ledger, keeping only each voter’s most recent, trust-cleared vote.

-- Exact reconciliation at poll close: replays the append-only vote ledger,
-- keeping only each voter's final choice, to produce the authoritative count
SELECT option_id, COUNT(*) AS exact_votes
FROM (
    SELECT DISTINCT ON (voter_token) voter_token, option_id
    FROM vote_ledger
    WHERE poll_id = $1
      AND trust_score >= 0.15
    ORDER BY voter_token, submitted_at DESC
) final_votes
GROUP BY option_id
ORDER BY exact_votes DESC;

This runs in O(n log n) for n ledger rows in the poll, dominated by the sort backing DISTINCT ON, and the (poll_id, voter_token, submitted_at DESC) index defined earlier lets Postgres satisfy that ordering directly from the index rather than sorting the whole table in memory.

Scaling and Performance

The Sharded Vote Counters and the WebSocket Broadcast Fan-out are the two components under the most direct load, and they scale along different axes: counters scale by adding shards per option, fan-out scales by adding pods and spreading connections across them with consistent hashing on viewer id.

A poll that unexpectedly goes viral partway through its window, a moment gets clipped and shared, a celebrity retweets it, can see vote volume jump by an order of magnitude with no warning. The shard count per option is not fixed at poll creation time; once a poll’s measured write rate crosses a threshold, it is automatically re-sharded to a higher shard count, the same fix a hot key gets anywhere else in a distributed system, split it finer rather than throwing more compute at the same partition. The fan-out tier scales independently and continuously, since connection count (viewers) and write volume (voters) are only loosely correlated, a poll can have five million viewers and a hundred thousand voters at the same time.

Capacity Estimation:

Given:
  - 1,000,000 votes submitted within a 30-second live voting window
  - Average vote payload: ~180 bytes (poll_id, option_id, voter_token,
    idempotency_key, signals)
  - 16 shards per option, 4 options per poll = 64 shard counters
  - Aggregator publish interval: 500ms
  - 5,000,000 concurrent viewer WebSocket connections watching the tally

Sustained vote throughput:
  1,000,000 votes / 30 sec = ~33,333 votes/sec average
  Peak burst (first/last few seconds of the window): ~100,000-120,000 votes/sec

Ingestion bandwidth:
  33,333 votes/sec * 180 bytes = ~6 MB/sec average inbound
  120,000 votes/sec * 180 bytes = ~21.6 MB/sec at peak burst

Shard write rate (per shard, evenly hashed):
  120,000 votes/sec / 64 shard counters = ~1,875 INCR ops/sec/shard at peak,
  comfortably inside a single Redis shard's single-digit-microsecond INCR cost

Aggregation reads:
  64 shard keys read via MGET every 500ms = 128 MGET ops/sec total,
  independent of vote volume, since aggregation cost scales with shard
  count, not vote count

Broadcast fan-out bandwidth:
  5,000,000 connections * ~40 bytes/diff message * 2 messages/sec
  (500ms cadence) = ~400 MB/sec aggregate egress across the fan-out tier,
  split across roughly 50 fan-out pods at 100,000 connections/pod
  = ~8 MB/sec egress per pod

The dominant bottleneck under normal load is not the aggregate numbers above, all of which are modest per shard and per pod, it is the transition moments: the first second of a window when every viewer’s client tries to submit at once, and the handful of seconds around a viral spike when a poll’s shard count is still catching up to its new write rate. Both are addressed the same way, detect the rate change fast and re-partition before latency degrades, rather than waiting for a latency SLA breach to trigger a response.

Failure Modes and Recovery

Failure	Detection	Impact	Recovery
Idempotency cache (Redis) evicts a key under memory pressure before a legitimate retry arrives	Ledger insert fails on the (poll_id, idempotency_key) unique constraint instead of the fast-path cache catching it	A retry briefly falls through to the slower defense layer instead of the fast one	The ledger’s unique constraint rejects the duplicate insert; the request is treated as already_accepted and no shard counter is touched twice
A vote counter shard’s process or node crashes	Health check / heartbeat failure on that shard’s process	The displayed live tally briefly undercounts that shard’s votes	The most recent checkpoint from shard_checkpoints restores an approximate value, and the gap fully self-heals at poll close, since the final tally is always recomputed exactly from vote_ledger, never from shard counters
A ballot-stuffing script floods votes from a single device or IP at superhuman velocity	Trust scoring detects abnormal submission velocity and datacenter/proxy signals in real time	Without mitigation, one option could receive a disproportionate share of votes	Score falls below TRUST_FLOOR; votes are shadow-filtered from both the live tally and the final reconciliation, but still logged in the ledger for audit, without alerting the sender
A WebSocket fan-out pod crashes mid-broadcast	Connection drop detected by the load balancer / client heartbeat	Viewers connected to that pod stop receiving tally updates	Client auto-reconnects to a healthy pod via the load balancer; the new pod immediately pulls the latest snapshot rather than waiting for the next 500ms tick, so the viewer catches up instantly
Clock skew between ingestion nodes near the poll’s closes_at boundary	Vote timestamps recorded with node-local clocks diverge from the poll’s authoritative close time	Votes arriving right at the boundary could be inconsistently included or excluded depending on which node processed them	Poll close is enforced by a single authoritative timestamp source rather than each node deciding locally, and a short grace buffer accepts votes slightly after closes_at into a late bucket resolved during reconciliation
The aggregator’s publish pipeline backs up during a viral spike	Publish queue depth or fan-out lag alert crosses a threshold	The displayed tally lags beyond the 2-second consistency window	The aggregator degrades from a 500ms to a longer interval under backpressure rather than falling further behind, and each fan-out pod independently caps its own egress rate

Comparison of Approaches

Approach	Latency	Complexity	Failure mode	Best fit
Single global atomic counter (UPDATE ... SET count = count + 1 under a row lock)	Fast at low volume, degrades sharply under contention	Low	Row-lock contention serializes every writer into one queue, becoming the bottleneck at high QPS	Small internal polls with a few hundred voters
Sharded in-memory counters with periodic aggregation (this design)	Low, bounded by the consistency window (about 2 seconds)	Medium	A crashed shard silently undercounts until checkpoint recovery or poll-close reconciliation	Large live-broadcast polls needing near-real-time display at high write volume
CRDT G-Counter replicated across regions	Low locally, eventually consistent globally	High	Merge overhead grows with node count, and a batch reconciliation step is still needed for an exact final number	Globally distributed audiences voting from multiple regions simultaneously
Fully synchronous durable counter (every vote a committed database transaction with strongly consistent reads)	Slow, write latency dominated by fsync and replication	Low-medium	Throughput is hard-capped by the durability path; a burst overwhelms the write path directly	Low-volume polls where exact real-time consistency matters more than raw scale, such as shareholder or governance votes
Append-only log plus stream processing recompute (Kafka feeding a continuous stream aggregation job)	Medium, bound by the stream processor’s checkpoint interval	High	Reprocessing lag under backpressure, plus the operational complexity of running a full streaming platform	Polls needing rich real-time analytics beyond a simple tally, such as demographic breakdowns or trending segments

We would pick sharded in-memory counters with periodic aggregation, backed by ledger-based exact reconciliation at close, for any broadcast-scale live poll, because it is the only approach here that keeps both halves of the requirement honest: a fast, bounded-staleness live number and an exact final one. A single global counter fails on throughput alone. A CRDT approach adds coordination complexity that single-region broadcasts do not need. A fully synchronous counter trades away exactly the write throughput a burst requires. A full streaming platform is overkill unless the product actually needs richer real-time analytics than a running tally.

Key Takeaways

• Idempotent submission and vote deduplication solve two different problems. Idempotency protects against a client retrying the same request; deduplication protects against the same voter submitting genuinely new requests more than once.
• Counter sharding turns a hot key into many cold ones by hashing each voter to exactly one shard, so concurrent writers for the same popular option never contend with each other.
• The live tally and the final tally are computed by two different mechanisms on purpose. The live number is an approximate, fast, periodically-aggregated snapshot; the final number is an exact replay of the durable, append-only vote ledger.
• Anti-stuffing works better as a continuous trust score than a binary block, since a hard threshold trains attackers to probe for the exact line while a continuous score punishes them for failing on any one of several independent signals.
• A bounded result consistency window (here, about 2 seconds) is a deliberate design choice, not a limitation to apologize for. Chasing per-vote exactness in the live number would have made the write path slower without making the poll more trustworthy to a viewer who can’t verify it live anyway.
• The aggregator decouples write volume from broadcast volume. Publishing on a fixed cadence rather than per vote keeps fan-out cost predictable regardless of how bursty the underlying vote rate gets.
• Hot polls need the same fix as any other hot shard: split finer, don’t just add more compute to the same partition.
• An append-only ledger, not an update-in-place counter, is what makes exact reconciliation possible after the fact.

The counter-intuitive lesson is that the number flashing on screen during the broadcast is deliberately allowed to be wrong within a bounded window. Exactness is pushed entirely to poll close, on purpose, because a viewer watching a live number has no way to verify it in the moment anyway, and spending engineering effort making the live number provably exact would have slowed down the one path, ingestion, that actually determines whether the system survives the burst.

Frequently Asked Questions

Q: Why not use one global counter with a lock instead of sharded counters?

A: A single counter serializes every writer behind one key, and at a peak of over 100,000 votes/sec that serialization becomes the bottleneck long before any individual operation’s raw cost does. Sharding turns one hot key into many independent cold ones, so writers for the same popular option never wait on each other, at the cost of needing a small aggregation step to read the total.

Q: Why not require login or accounts to prevent ballot stuffing?

A: Requiring an account before voting adds friction that most live-broadcast audiences won’t tolerate for a thirty-second window, and it doesn’t actually prevent stuffing, an attacker can script account creation just as easily as they can script anonymous votes. Device and behavioral signals, scored continuously, catch scripted abuse without adding a barrier that costs real participation from honest viewers.

Q: How does the system prevent someone from voting twice using two different devices?

A: It doesn’t, fully, and that’s a deliberate scope boundary rather than an oversight. Without accounts, there is no durable identity to tie two physical devices together with certainty; the system optimizes for making single-device, scripted stuffing expensive and detectable, not for perfectly attributing every vote to one human across every device they own. This tradeoff is explicit in the non-functional requirement to resist a 50x single-source attack, not a cross-device one.

Q: What happens if the trust score wrongly flags a legitimate, fast voter as suspicious?

A: Because the score is continuous rather than a hard block, a borderline-fast legitimate vote is more likely to land just above the trust floor than to be excluded outright, and unlike a hard rule, nothing in the response tells the voter anything happened differently. The floor is tuned conservatively specifically to bias toward false negatives (missing some real abuse) over false positives (silently dropping real votes), since the reputational cost of visibly under-counting real votes is higher than the cost of a small amount of undetected low-effort abuse.

Q: Why publish tally updates every 500ms instead of pushing every vote immediately?

A: Pushing every vote as its own broadcast message ties broadcast volume directly to vote volume, and at 100,000 votes/sec fanned out to 5,000,000 viewers, that amplification is not sustainable by any fan-out tier. A fixed publish cadence decouples the two, turning an unbounded, bursty rate into a fixed, predictable one, at the cost of the bounded staleness window the non-functional requirements already budget for.

Q: How is the final result guaranteed to be exact if the live number shown during the poll was only approximate?

A: The live number and the final number are computed by entirely different code paths reading from entirely different sources. The live number sums sharded, in-memory counters on a timer; the final number replays the durable, append-only vote ledger once, applying the same trust floor and last-vote-wins rule precisely rather than approximately. The two numbers can disagree slightly while the poll is open; only the ledger replay ships as the official result.

Interview Questions

Q: Walk through what happens, end to end, when a viewer taps a poll option and the tally on their own screen updates a moment later.

Expected depth: Cover the idempotency key check at the gateway, identity resolution and trust scoring, the hash-to-shard increment, the durable ledger insert, the aggregator’s next scheduled sum across all shards, and the fan-out pod computing a diff and pushing it to that viewer’s own connection, and why none of these steps happen synchronously in response to the tap itself past the initial acknowledgment.

Q: How would you detect and respond to a poll unexpectedly going viral mid-broadcast, where vote volume jumps by 10x with no warning?

Expected depth: Discuss monitoring per-poll write rate in real time, the threshold that triggers increasing shard count for that poll’s counters, why re-sharding mid-poll requires a migration strategy for in-flight counts (not just increasing N going forward), and how the WebSocket fan-out tier scales independently since viewer count and voter count aren’t the same number.

Q: Design the anti-stuffing trust scoring function. What signals would you include, and how would you avoid it becoming a binary block an attacker can simply probe against?

Expected depth: Cover combining multiple independent, weakly-correlated signals (submission velocity, IP/ASN reputation, connection age, device fingerprint reuse) multiplicatively into a continuous score, why a continuous floor resists probing better than a hard rule, and the tradeoff between false positives (dropping real votes) and false negatives (missing scripted abuse).

Q: Why does this design compute the live tally and the final tally using two different mechanisms instead of one? What would go wrong if you tried to make the live number always exactly match the eventual final number?

Expected depth: Discuss the lambda-architecture-style split between a fast, approximate speed layer (sharded counters, periodic aggregation) and an exact batch layer (ledger replay at close), and why forcing per-vote exactness into the live path would require exactly the kind of cross-writer coordination that sharding exists to avoid.

Q: How would you extend this design to support many concurrent independent polls running across different live broadcasts at the same time, rather than one poll at a time?

Expected depth: Cover partitioning shard counters, ledger writes, and aggregator instances by poll_id, why each poll’s aggregator can run as an independent per-poll leader rather than one global aggregator, and how the WebSocket fan-out tier’s consistent hashing needs to account for viewers subscribing to different polls without one poll’s viral spike starving another poll’s broadcast bandwidth.