Imagine running a gas station with a broken pump display. You need to show customers exactly how much gas is flowing - but the sensor inside the pump takes a reading only once a minute and the display lags by another 30 seconds. Meanwhile, there’s a guy with a garden hose pretending to fill up without paying. That is precisely the problem with view counts at YouTube scale: you want an accurate number fast, and you need to filter out fraud, but the mechanism that serves the number must never slow down the thing people actually care about - watching the video.

YouTube ingests roughly 5 billion views per day, or about 58,000 view events per second at peak. A naive counter - increment a single row in Postgres every time someone watches - would melt under this load. At 58K writes per second a single row would be the hottest spot in any database ever deployed. Even Redis, which is fast, would saturate a single key at somewhere around 100K-200K writes per second before CPU becomes the bottleneck - and that is with no concurrency protection. More critically, a single counter has no isolation: if the counting service restarts, you lose uncounted views. If the bot detection step is synchronous, it blocks video delivery. If you query the raw counter directly from video pages, count reads add latency to the most traffic-heavy path in the system.

The real challenge is that four concerns pull in opposite directions. Accuracy requires counting every real view exactly once. Latency requires never touching the counting path from the video serve path. Freshness requires counts that update in seconds for trending content. Fraud resistance requires discarding bot traffic without allowing bots to cause legitimate views to be lost. Getting all four right simultaneously is the engineering problem.

We need to solve for write throughput through counter sharding, approximate deduplication via HyperLogLog, asynchronous bot filtering off the critical path, and eventual consistency between in-memory counters and the durable source of truth. Let’s build it.

Requirements and Constraints

Functional Requirements

• Track a view when a user watches a video for at least 30 seconds (YouTube’s documented threshold)
• Display the current view count on the video watch page, updated within 30 seconds for normal videos
• For trending videos (top 0.1% by current velocity), update the displayed count within 5 seconds
• Resist inflation from bots, page refreshes within short windows, and automated crawlers
• Provide an admin API to query raw, unfiltered event counts for abuse investigation
• Support a “unique viewers” metric distinct from total play count (one viewer watching 10 times = 10 plays, 1 unique)

Non-Functional Requirements

• Write throughput: 58,000 view events per second sustained, 200,000 at peak (viral video moments)
• Read latency: View count reads on video pages must complete in under 10ms (P99)
• Count freshness: Less than 30 seconds lag for standard videos; less than 5 seconds for trending
• Durability: No view event lost once acknowledged by the edge layer, even if internal services crash
• Availability: 99.99% - view counting outages cause visible stale numbers, which erode creator trust
• Storage: ~800 million active videos; storing sharded counters + reconciliation records = roughly 200GB in Redis, 50GB in Postgres

Constraints and Assumptions

• The video serve path (CDN delivery) must be completely decoupled from counting; a counting outage must not affect playback
• View events are submitted via fire-and-forget beacons from the CDN edge, not from the browser directly
• We accept eventual consistency: the displayed count can be up to 30 seconds stale in the normal case
• Bot filtering runs asynchronously - some bot views may be counted briefly before the filter catches up, but reconciliation corrects this within hours
• “View” definition is a product decision fixed upstream; we design for that threshold, not for detecting it

High-Level Architecture

The system splits cleanly into a write path and a read path, with an async reconciliation job connecting them to the durable store.

The write path starts at the CDN edge. When the player’s 30-second watch threshold triggers, the client fires a beacon to the nearest CDN edge node. The edge node enqueues the event to Kafka (the view_events topic, 30 partitions) and immediately returns a 200 - it never touches a database. Kafka gives us durability: the event is on disk before we ACK.

The Bot Filter Service consumes from the view_events topic, scores each event using IP velocity, user-agent signals, watch duration, and replay rate, and emits valid events onto a filtered clean_view_events topic. The View Aggregation Service consumes from clean_view_events, batches events in 500ms windows, and writes increments to Redis Counter Shards and HyperLogLog registers.

The read path is completely separate. The Count Read API serves GET /count/{video_id} requests from video pages. It reads from a Redis read cache with a 30-second TTL (5 seconds for trending videos). It never reads from the write path’s shards directly.

An hourly Reconciliation Job sums all shard counters for each video and writes a canonical count to Postgres, which becomes the source of truth for historical reporting, analytics, and the fallback when Redis is unavailable.

The Ingestion Pipeline

The ingestion pipeline’s job is to durably capture every view event off the critical path, at any volume, without acknowledgment delays.

The CDN edge node does exactly two things when a view event arrives: enqueue to Kafka and return 200. The Kafka producer is configured with acks=1 (leader acknowledgment only) and linger.ms=5 to batch events from concurrent watchers into fewer produce requests. The tradeoff: acks=1 risks losing an event if the Kafka leader crashes before replication. At YouTube scale this is acceptable - losing 0.001% of events in a node crash scenario is better than adding latency to 100% of video deliveries.

# View event beacon handler on the CDN edge (Python/FastAPI sketch)
# Demonstrates: fire-and-forget event submission, no database touch
from aiokafka import AIOKafkaProducer
import msgpack, time

producer = AIOKafkaProducer(
    bootstrap_servers="kafka-cluster:9092",
    acks=1,
    linger_ms=5,
    compression_type="lz4",
)

async def handle_view_event(video_id: str, viewer_id: str, ip: str, ua: str):
    event = {
        "video_id": video_id,
        "viewer_id": viewer_id,
        "ip": ip,
        "ua": ua,
        "ts": int(time.time() * 1000),
        "session_id": generate_session_id(viewer_id, video_id),
    }
    # Partition by video_id so events for the same video land on the same partition
    key = video_id.encode()
    await producer.send("view_events", value=msgpack.packb(event), key=key)
    return {"status": "ok"}  # never block on Kafka confirmation

Partitioning by video_id means all events for a given video land on the same Kafka partition, which preserves ordering and makes downstream deduplication simpler. With 30 partitions and 800 million videos, roughly 27 million videos per partition - all well within Kafka’s limits.