Think of the trending topics sidebar as a live newspaper editor standing in a room with 500,000 people all talking at once. The editor’s job is not to transcribe everything - it’s to notice which words suddenly appear in 10,000 conversations simultaneously when five minutes ago nobody was saying them. That pattern-detection problem, at machine speed and global scale, is what Twitter’s trending pipeline solves.

The naive approach has two faces, and both fail. The first face: store every tweet’s tokens, run GROUP BY token ORDER BY COUNT DESC every 60 seconds. At 500,000 tweets per second with an average of 8 tokens per tweet, that’s 4 million token events per second flowing into a database. After one hour you have 14.4 billion rows to scan. The query takes minutes, not seconds. The second face: discard old data and only track the last 5 minutes exactly. This eliminates the scan cost but requires updating a counter for every token in every tweet - that’s still 4 million counter increments per second against a shared key-value store. Redis can handle roughly 1 million operations per second on a single node. The math doesn’t work.

What makes this problem genuinely hard is the combination of three forces pulling in opposite directions. You need velocity (a topic exploding in the last 5 minutes) rather than volume (a topic that has been consistently popular for months - “good morning” trends every day, and nobody wants to see that). You need recency without full-history replay. And you need geographic accuracy - what’s trending in Brazil is different from what’s trending in Japan, and showing one region’s trends to another degrades the product severely.

We need to solve for approximate counting with bounded error, sliding window aggregation without replaying history, geographic partitioning without losing cross-region signals, and bot suppression before counts pollute the trend signal.

Requirements and Constraints

Functional Requirements

• Detect the top-30 trending topics per geographic region and worldwide
• Refresh trends every 60 seconds
• Support at least 20 distinct geographic regions (US-East, US-West, EU-West, APAC, LATAM, etc.)
• A “trend” is a topic whose mention velocity (rate of change) exceeds a threshold, not just raw volume
• Filter out known bot accounts and coordinated inauthentic behavior before computing trends
• Suppress recurring daily topics that trend every day at the same time (not genuinely emergent)

Non-Functional Requirements

• Ingest throughput: 500,000 tweets per second peak (300,000 average)
• Trend refresh latency: less than 60 seconds from tweet to visible trend
• Serving latency: trends API returns in under 50ms (p99)
• Availability: 99.9% uptime (8.7 hours of downtime per year)
• Eventual consistency acceptable: a topic that goes viral can appear in trends within 2 refresh cycles (120 seconds max)
• Memory budget per processing node: under 512MB for all in-memory sketch state
• Zero reprocessing of historical data per 60-second cycle

Constraints and Assumptions

• We do not need to retroactively recalculate trends (no historical replay)
• Trend lists are read-heavy (millions of readers) versus write-heavy (one writer per region per 60s)
• Geographic assignment uses IP geolocation and account home region - we assume this is pre-computed upstream
• Tweet language detection is handled upstream; trends are per-region but language-specific filtering is a separate concern
• We are not building the tweet storage or fanout system - only the trending computation layer

High-Level Architecture

The pipeline has four layers stacked vertically: ingest, stream processing, aggregation, and serving. Data flows downward; control signals flow upward.

Tweet Stream feeds raw tweet events into a Kafka cluster partitioned by region_id. Kafka acts as the durable buffer between unpredictable ingest spikes and the downstream processing system. Each partition is owned by a dedicated Flink worker that maintains a sliding window and a Count-Min Sketch in memory. The Flink workers emit their local top-K lists every 60 seconds. A Regional Aggregator merges the outputs from all workers in a region into a single ranked list. That list is written to a Redis sorted set keyed by region. The Trends API reads from Redis and serves it through a CDN edge cache with a 30-second TTL.

Bot Filter sits inline on the Kafka consumer side. Each Flink worker applies velocity-based bot detection before a tweet’s tokens are counted. This keeps bot-inflated counts from ever entering the sketch.

Velocity Scorer applies a decay function and computes velocity (rate of change relative to baseline) as each window emits. This is what distinguishes a genuine emerging trend from a chronically popular topic.

The Ingest and Partitioning Layer

The ingest layer’s job is to route each tweet to the right Flink worker without creating hot partitions. This sounds like a solved problem until you consider regional imbalances: the APAC partition sees a massive spike every morning when India’s commute hour begins, while the LATAM partition sees relatively low volume during the same UTC window.

Kafka partitions are assigned by hash(region_id) mod num_partitions. Within a region, tweets are distributed across multiple partitions using hash(user_id). This two-level partitioning serves two goals: region isolation (so APAC traffic cannot crowd out EU traffic) and within-region parallelism (so one region uses multiple Flink workers, each seeing a consistent slice of that region’s users).

# Kafka topic configuration for tweet ingest
# topic: tweet-tokens-v2
bootstrap.servers: kafka-cluster:9092
num.partitions: 120
replication.factor: 3
retention.ms: 3600000  # 1 hour retention - enough for reprocessing after a worker crash
min.insync.replicas: 2
compression.type: lz4
# Partition assignment: first 20 partitions per region (6 regions = 120 total)
# Producer key: "{region_id}:{hash(user_id) mod 20}"