Imagine you are the librarian for a collection of a billion photographs, responsible for handing every visitor a personally curated stack the moment they walk through the door - before they have even told you what they want. Now imagine you have 2 billion visitors per day, each staying only 30 seconds if the first page does not captivate them, and half your collection was uploaded in the last 24 hours. That is the engineering problem behind Instagram Explore.

The naive approach - score every post in the catalog against every user profile - collapses immediately. 2 billion users times 100 billion pieces of content equals 200 quintillion pairs. Even a microsecond comparison per pair would take more compute than exists on Earth. The solution is a staged funnel that discards 99.99% of candidates before any expensive model sees them, preserves diversity so the grid does not collapse into an echo chamber, and injects trending signals fast enough that a post going viral appears on relevant Explore grids within minutes rather than hours.

Three tensions define every decision in this system. Relevance versus freshness: the most personalized recommendations come from a model trained on weeks of engagement history, but content that went viral three minutes ago has no engagement history at all. Personalization versus diversity: a pure relevance model converges on a narrow slice of content that matches past behavior, suffocating discovery and accelerating user churn. Latency versus depth: deeper neural re-ranking produces better results but burns more compute; shallow retrieval is fast but imprecise.

We need to solve for candidate retrieval at billion-user scale, embedding freshness under content churn, real-time trend injection, and sub-200ms end-to-end latency simultaneously.

Requirements and Constraints

Functional Requirements

• Generate a personalized Explore grid of 100-200 posts for any user on demand
• Refresh recommendations as the user scrolls (pagination via cursor)
• Surface content from accounts the user does not follow
• Detect trending content and inject it into relevant grids within 5 minutes of a trend breaking
• Respect content policy: exclude blocked accounts, reported posts, restricted categories per user settings

Non-Functional Requirements

• 2 billion users, ~400 million daily active Explore sessions
• Peak: ~5 million grid requests per second during prime time
• End-to-end latency: P99 under 200ms, P50 under 80ms
• Recall@100 (top-100 retrieved items contain at least one that will be engaged): >60%
• Embedding freshness: new content indexed within 60 seconds of upload
• Trend detection: trending posts appear in relevant grids within 5 minutes
• 99.99% availability (Explore is a major engagement surface)

Constraints

• Cold-start users (new signups) receive a popularity-based fallback grid
• Content from private accounts is excluded unless the user follows them
• We do not build a real-time bidding or ads layer here - that is a separate system
• Geographic content restrictions enforced at retrieval time, not re-ranking

High-Level Architecture

The system is a four-stage funnel: candidate retrieval, lightweight ranking, neural re-ranking, and post-processing (diversity + trend injection). Surrounding this funnel are two supporting systems: an embedding pipeline that keeps content and user vectors fresh, and a trend detection pipeline that feeds viral signals into the top of the funnel.

A user opens Explore. The request hits an API gateway that authenticates the session and routes to a recommendation orchestrator. The orchestrator fans out to a candidate retrieval service which runs approximate nearest-neighbor (ANN) search against a 100-billion-vector index, returning 2,000-5,000 candidates. A lightweight ranker - a linear model running in under 10ms - scores these candidates and cuts the list to 500. A neural re-ranker - a two-tower deep model - re-scores the top 500 and cuts to 200. Finally a post-processor applies diversity filtering, injects trending posts, enforces blocks and restrictions, and returns the final grid.

The Embedding Pipeline

The embedding pipeline is the engine that converts raw content into the vectors the retrieval stage searches. Think of it as a translation service: raw pixels, captions, and engagement counts become points in a shared 256-dimensional space where semantic similarity corresponds to proximity.

When a post is uploaded, a media processing service extracts a visual embedding using a vision transformer (ViT) fine-tuned on Instagram’s content distribution. A text encoder processes the caption and hashtags. An engagement encoder computes early engagement signals (saves-to-impressions ratio, share velocity) as a sparse feature vector. These three signals are concatenated and projected through a learned fusion layer to produce the final 256-dimensional content embedding.