Imagine you run a video store with 15,000 titles and 280 million customers walk through the door each day. Each customer should see a different arrangement of titles on the shelves - an arrangement tuned to their specific tastes, current mood, and what they watched last Tuesday. In a physical store, you could hire 280 million personal shoppers. In a software system, you need to solve this in under 100 milliseconds per request, for users arriving at tens of thousands per second, with recommendations that meaningfully update within hours of a user’s last viewing session.

Netflix reports that 80% of the hours streamed on its platform come from recommendations rather than direct search. The homepage - that grid of rows and tiles a user sees when they open the app - is the primary revenue mechanism of the entire business. Get the recommendations wrong and users spend too long browsing, feel the platform doesn’t “get” them, and eventually churn. The homepage is not just a UI; it is a real-time personalization system that generates approximately 280 million unique documents per day, each one a ranked list of 40-80 titles selected from a catalog of 15,000.

The engineering tension is three-way. Freshness vs. stability: a user who watched three episodes of a Korean thriller last night should see Korea-adjacent content on the homepage this morning - but a model trained on data from 6 hours ago has not yet incorporated last night’s session. Accuracy vs. scalability: the most accurate recommendation approach involves running a separate model instance per user - that scales linearly with user count, which is impossible at 280 million. Personalization vs. cold start: a brand new user with zero history has no signal for a collaborative filtering model - but showing them generic popularity-ranked titles destroys the promise of personalization.

We need to solve for three architectural requirements simultaneously: offline model training that produces high-quality user and item representations refreshed every few hours using matrix factorization; near-real-time feature serving that incorporates last night’s sessions into today’s recommendations without a full model retrain; and an online serving layer that retrieves and ranks candidates in under 100ms at query time. The glue between offline accuracy and online speed is the embedding store - a compact numerical representation of each user’s taste that can be looked up in microseconds and used to query an approximate nearest-neighbor index for candidates.

Requirements and Constraints

Functional Requirements

• Generate a unique, personalized homepage for each of 280 million active users
• Update homepage recommendations to reflect new viewing behavior within 2-6 hours
• Return homepage recommendations within 100ms p99 latency
• Handle cold start for new users (no watch history) and new titles (no view data)
• Support A/B testing of multiple recommendation algorithms simultaneously across user buckets
• Provide diverse recommendations: not just one genre or one franchise dominating the homepage
• Support multiple recommendation row types: “Continue Watching”, “Top Picks for You”, “Trending Now”, “Because You Watched X”

Non-Functional Requirements

• 280 million active users, peak of 30 million concurrent homepage loads (primetime)
• At 30M concurrent loads, serving throughput: ~300,000 homepage requests per second
• Catalog size: 15,000+ titles globally (varies by region)
• Training data volume: approximately 10 billion implicit rating events per day
• Model size: user embedding matrix 280M x 128 dimensions = approximately 140 GB; item matrix 15K x 128 = ~7.5 MB
• Model freshness: user profile updates within 1 hour of viewing event; full model retrain every 1-6 hours
• Recommendation quality: play rate (% of homepage tiles that get clicked and watched) target 30%+ for top-10 recommendations

Constraints and Assumptions

• We use implicit feedback (watch percentage, re-watch, search after browse) not explicit ratings (thumbs)
• Personalization is per-user-account, not per-profile within an account (sub-profile personalization is a separate problem)
• We focus on the content recommendation engine; thumbnail personalization (A/B testing artwork) is a separate system
• We assume the catalog metadata (genre, cast, synopsis, language) is pre-processed and available in the Feature Store
• Real-time inventory (content availability by country) is handled upstream; we recommend only available titles

High-Level Architecture

The system decomposes into five major subsystems: the Event Ingestion Pipeline (captures raw user interactions), the Offline Training Pipeline (produces user and item embeddings via matrix factorization), the Feature Store (serves precomputed features to both training and serving), the Candidate Generation and Ranking Service (the online serving layer), and the A/B Testing Framework (controls model rollout and experiment allocation).

User interactions - play events, pauses, searches, browse hover durations - are published to a Kafka event log in near-real-time. A Flink stream processor consumes this log and performs two jobs: enriching events with catalog metadata (joining the event title_id against the Feature Store to get genre, language, etc.) and updating lightweight user preference vectors in Redis. These lightweight “near-real-time” vectors capture recency signal (what the user watched in the last 24 hours) and complement the heavier user embeddings from the offline training pipeline.

The offline training pipeline runs on a Spark cluster, typically every 1-6 hours. It reads from the full 7-day event log stored in S3, constructs an implicit rating matrix, and runs Alternating Least Squares (ALS) matrix factorization to produce 128-dimensional user and item vectors. These vectors are written to a distributed embedding store (Cassandra with Redis cache). Separately, item vectors are loaded into a FAISS Approximate Nearest Neighbor index on the serving nodes.

When a user opens the Netflix app, the homepage request arrives at the Recommendation Service. The service fetches the user’s 128-dimensional vector from the embedding store (Redis, ~0.5ms), queries the FAISS index to retrieve the top-500 candidate titles nearest to the user vector (~5ms), fetches contextual features (device type, time of day, previously watched filter list) from the Feature Store (~2ms), runs a lightweight XGBoost ranking model over the 500 candidates to produce final scores (~10ms), applies diversity and watched-content filters (~1ms), and assembles the homepage rows (~1ms). Total serving time: approximately 20-25ms, well within the 100ms SLA.

Matrix Factorization and Collaborative Filtering

Collaborative filtering is the insight that users with similar viewing history have similar tastes, and that this similarity can be exploited without ever knowing why the two users have similar taste. If User A and User B both completed 5 episodes of “Stranger Things”, “Dark”, and “Money Heist”, then we can reasonably infer that a title User A watched but User B hasn’t is a good recommendation for User B - even if we don’t know that all three watched titles happen to be tense, plot-driven thriller series.