Seismologists do not wait for a fault to produce a magnitude-seven quake before they pay attention. They watch for the rate at which small tremors are accelerating, because a swarm of minor shocks speeding up is a leading indicator of something much bigger, long before the cumulative shaking crosses any dramatic threshold. A hashtag trending system has to make the same trade. Absolute post volume tells you what is already big, but by the time a hashtag has genuinely large volume, the breaking news, meme, or match that is driving it has often already peaked. Velocity, how fast the volume is climbing right now relative to what is normal for that tag at that place and time, is what tells you something is starting.

Now scope that to geography. Trending is not a single global phenomenon wearing a country filter. A hashtag about a domestic cricket match trending hard in Mumbai means nothing to a reader in Sao Paulo, and forcing every region to share one worldwide chart buries a genuinely large regional breakout under whatever happens to be biggest across the whole platform that hour. The system has to compute an independent ranking per region, the way a chain of regional radio stations each keeps its own top-40 chart instead of broadcasting one national countdown to every city regardless of what is actually popular there.

The naive approach, keep one running counter per (region, hashtag) pair and re-sort the list every minute, fails for reasons that only show up at scale. First, it treats volume as the whole signal, so a tag that spikes to the same predictable size every single Monday morning, #MondayMotivation, #ThrowbackThursday, permanently occupies a trending slot that a genuinely surprising story deserves more. Second, keeping an exact counter for every hashtag that has ever appeared is a cardinality problem, not a compute problem: the long tail of one-off tags, typos, and case or unicode variants of the same tag is effectively unbounded, and a hash map that grows without limit eventually exhausts the memory of whatever process is holding it. Third, a live counter has no concept of “early.” By the time a tag’s raw count is large enough to rank in a naive top-K, most of its growth has already happened, which is exactly the failure mode the seismologist is built to avoid.

We need to solve for three things simultaneously: partition the ingestion firehose by region so that one region’s volume can never starve or slow down another region’s freshness, maintain sliding-window counts of hashtag occurrence cheaply despite a cardinality that is effectively unbounded, and score by velocity against each hashtag’s own historical baseline, not a single flat threshold, so a recurring ritual tag gets dampened while a genuine breakout surfaces before it is large enough to win on volume alone.

Requirements and Constraints

Functional Requirements

• Extract every hashtag from every post as it is created, in near real time, not on a delayed batch cycle
• Normalize each raw hashtag string to a canonical form so casing, unicode variants, and formatting differences never fragment the same trend into separate counters
• Resolve each post to exactly one geographic region based on the author’s location signal before it enters aggregation
• Maintain a sliding-window occurrence count of every hashtag, per region, refreshed at least once a minute
• Compute a composite score per (region, hashtag) pair that blends current volume with recent velocity, so a fast-accelerating tag can outrank a merely large one
• Detect and separately surface emerging breakout tags, ones whose velocity is anomalously high relative to their own history even while their absolute volume is still below the region’s top ranks
• Dampen hashtags that spike on a predictable recurring schedule so they do not permanently occupy trending slots that a genuine surprise deserves
• Maintain and serve a ranked top-K trending list per region, refreshed on the same one-minute cadence as the underlying windows
• Support hundreds of independently sized regions, from a single country down to a metro area inside a large country, without one region’s volume distorting another region’s list

Non-Functional Requirements

• Scale: roughly 500 million posts per day platform-wide; about 32% contain at least one hashtag, and tagged posts carry an average of 1.7 hashtags, producing around 272 million hashtag instances per day
• Regions: approximately 600 tracked regions, a mix of whole countries and metro-level breakdowns inside the handful of countries large enough to need them
• Freshness: every region’s trending list must reflect activity from within the last 60 to 90 seconds
• Burst tolerance: a single region can spike to 8x its own baseline volume during a national event without degrading the freshness of any other region
• Cardinality: tens of millions of distinct hashtags appear across a single day; counting must stay within a fixed, bounded memory budget per region rather than growing with the number of distinct tags ever seen
• Read latency: the trending list read path targets a p99 under 100ms, served from a precomputed snapshot, never a live aggregation query
• Availability: the serving layer targets 99.9% uptime; the aggregation pipeline’s health in one region must never block or slow down another region’s pipeline
• Approximation tolerance: counts within roughly 1% relative error are acceptable in exchange for bounded memory; the system deliberately trades a small, quantifiable amount of counting precision for a fixed cost that does not grow with cardinality

Constraints and Assumptions

• Post storage and retrieval, hashtag search, click-through analytics, and any ad-targeting built on top of trending data are out of scope; this design only decides what is trending and where
• We assume an upstream service resolves each post’s author to a region_id from a geo-IP or declared-locale signal; this design consumes that resolution, it does not compute it
• We assume partial spam and bot filtering happens upstream before a post reaches this pipeline, but the aggregation layer still needs its own defenses against residual coordinated flooding, since no upstream filter catches everything
• The region taxonomy, roughly 600 fixed regions, is managed by a separate reference-data service; this design treats the set of valid region_id values as a slowly changing input, not something it computes
• UI ranking, personalization of the trending widget, and localization of hashtag display text are handled by the client and are out of scope here

High-Level Architecture

The system has five major components: the Ingestion, Extraction and Normalization Layer, the Per-Region Stream Partitioning Layer, the Sliding Window Aggregation Engine, the Velocity and Breakout Scoring Engine, and the Top-K Materialization and Serving Layer.

The Ingestion, Extraction and Normalization Layer is the front door for every post: it pulls out raw hashtag substrings, canonicalizes each one so that trivial formatting differences collapse into a single counter, and attaches the resolved region_id. The Per-Region Stream Partitioning Layer routes each normalized hashtag event onto a partition keyed by region, so that a hot region’s volume is physically isolated from every other region’s processing capacity. The Sliding Window Aggregation Engine is the computational core, maintaining a bounded-memory, approximate occurrence count for every hashtag inside each region’s rolling window. The Velocity and Breakout Scoring Engine reads each window’s counts alongside a per-hashtag seasonal baseline and produces a composite score that separates genuine breakouts from predictable recurring spikes. The Top-K Materialization and Serving Layer merges scored candidates into a ranked list per region and answers read requests from a precomputed, cached snapshot rather than from live aggregation state.

A hashtag’s life inside this system looks like this: a post is created, its text is scanned for # tokens, each token is normalized and paired with the post’s resolved region, and that event lands on the region’s stream partition within milliseconds. Once a minute, the aggregation engine closes the current one-minute bucket, folds it into a trailing ten-minute sliding window, and hands the region’s updated counts to the scoring engine. The scoring engine looks up each hashtag’s short-term and seasonal baselines, computes a velocity-aware composite score, applies a dampening multiplier to tags whose spike matches their own predictable historical pattern, and flags tags whose velocity is anomalous relative to their history as emerging even if their raw count is still small. The top-K materializer merges scored candidates across any shards a busy region has been split into, writes the new ranked list to the trending store, and the serving layer’s cache picks it up on the next client read.

Component Deep Dives

The Ingestion, Extraction and Normalization Layer

This component’s job is to turn raw post text into a clean, canonical, region-tagged hashtag event before anything downstream ever counts it.

The mistake a simpler design makes is treating hashtag text as already clean. Real posts produce #WorldCup, #worldcup, #WORLDCUP, and a version with a trailing zero-width joiner pasted in from a copied caption, and if all four are counted as distinct keys, a single trend gets fragmented into four small counters, none of which is large enough to rank, while the tag itself is genuinely huge. Normalization has to run exactly once, at ingestion, so every downstream counter, sketch, and baseline lookup agrees on one canonical key for the same trend.