Imagine you run a video store with 15,000 titles and 280 million customers walk through the door each day. Each customer should see a different arrangement of titles on the shelves - an arrangement tuned to their specific tastes, current mood, and what they watched last Tuesday. In a physical store, you could hire 280 million personal shoppers. In a software system, you need to solve this in under 100 milliseconds per request, for users arriving at tens of thousands per second, with recommendations that meaningfully update within hours of a user’s last viewing session.

Netflix reports that 80% of the hours streamed on its platform come from recommendations rather than direct search. The homepage - that grid of rows and tiles a user sees when they open the app - is the primary revenue mechanism of the entire business. Get the recommendations wrong and users spend too long browsing, feel the platform doesn’t “get” them, and eventually churn. The homepage is not just a UI; it is a real-time personalization system that generates approximately 280 million unique documents per day, each one a ranked list of 40-80 titles selected from a catalog of 15,000.

The engineering tension is three-way. Freshness vs. stability: a user who watched three episodes of a Korean thriller last night should see Korea-adjacent content on the homepage this morning - but a model trained on data from 6 hours ago has not yet incorporated last night’s session. Accuracy vs. scalability: the most accurate recommendation approach involves running a separate model instance per user - that scales linearly with user count, which is impossible at 280 million. Personalization vs. cold start: a brand new user with zero history has no signal for a collaborative filtering model - but showing them generic popularity-ranked titles destroys the promise of personalization.

We need to solve for three architectural requirements simultaneously: offline model training that produces high-quality user and item representations refreshed every few hours using matrix factorization; near-real-time feature serving that incorporates last night’s sessions into today’s recommendations without a full model retrain; and an online serving layer that retrieves and ranks candidates in under 100ms at query time. The glue between offline accuracy and online speed is the embedding store - a compact numerical representation of each user’s taste that can be looked up in microseconds and used to query an approximate nearest-neighbor index for candidates.

Requirements and Constraints

Functional Requirements

• Generate a unique, personalized homepage for each of 280 million active users
• Update homepage recommendations to reflect new viewing behavior within 2-6 hours
• Return homepage recommendations within 100ms p99 latency
• Handle cold start for new users (no watch history) and new titles (no view data)
• Support A/B testing of multiple recommendation algorithms simultaneously across user buckets
• Provide diverse recommendations: not just one genre or one franchise dominating the homepage
• Support multiple recommendation row types: “Continue Watching”, “Top Picks for You”, “Trending Now”, “Because You Watched X”

Non-Functional Requirements

• 280 million active users, peak of 30 million concurrent homepage loads (primetime)
• At 30M concurrent loads, serving throughput: ~300,000 homepage requests per second
• Catalog size: 15,000+ titles globally (varies by region)
• Training data volume: approximately 10 billion implicit rating events per day
• Model size: user embedding matrix 280M x 128 dimensions = approximately 140 GB; item matrix 15K x 128 = ~7.5 MB
• Model freshness: user profile updates within 1 hour of viewing event; full model retrain every 1-6 hours
• Recommendation quality: play rate (% of homepage tiles that get clicked and watched) target 30%+ for top-10 recommendations

Constraints and Assumptions

• We use implicit feedback (watch percentage, re-watch, search after browse) not explicit ratings (thumbs)
• Personalization is per-user-account, not per-profile within an account (sub-profile personalization is a separate problem)
• We focus on the content recommendation engine; thumbnail personalization (A/B testing artwork) is a separate system
• We assume the catalog metadata (genre, cast, synopsis, language) is pre-processed and available in the Feature Store
• Real-time inventory (content availability by country) is handled upstream; we recommend only available titles

High-Level Architecture

The system decomposes into five major subsystems: the Event Ingestion Pipeline (captures raw user interactions), the Offline Training Pipeline (produces user and item embeddings via matrix factorization), the Feature Store (serves precomputed features to both training and serving), the Candidate Generation and Ranking Service (the online serving layer), and the A/B Testing Framework (controls model rollout and experiment allocation).

User interactions - play events, pauses, searches, browse hover durations - are published to a Kafka event log in near-real-time. A Flink stream processor consumes this log and performs two jobs: enriching events with catalog metadata (joining the event title_id against the Feature Store to get genre, language, etc.) and updating lightweight user preference vectors in Redis. These lightweight “near-real-time” vectors capture recency signal (what the user watched in the last 24 hours) and complement the heavier user embeddings from the offline training pipeline.

The offline training pipeline runs on a Spark cluster, typically every 1-6 hours. It reads from the full 7-day event log stored in S3, constructs an implicit rating matrix, and runs Alternating Least Squares (ALS) matrix factorization to produce 128-dimensional user and item vectors. These vectors are written to a distributed embedding store (Cassandra with Redis cache). Separately, item vectors are loaded into a FAISS Approximate Nearest Neighbor index on the serving nodes.

When a user opens the Netflix app, the homepage request arrives at the Recommendation Service. The service fetches the user’s 128-dimensional vector from the embedding store (Redis, ~0.5ms), queries the FAISS index to retrieve the top-500 candidate titles nearest to the user vector (~5ms), fetches contextual features (device type, time of day, previously watched filter list) from the Feature Store (~2ms), runs a lightweight XGBoost ranking model over the 500 candidates to produce final scores (~10ms), applies diversity and watched-content filters (~1ms), and assembles the homepage rows (~1ms). Total serving time: approximately 20-25ms, well within the 100ms SLA.

Matrix Factorization and Collaborative Filtering

Collaborative filtering is the insight that users with similar viewing history have similar tastes, and that this similarity can be exploited without ever knowing why the two users have similar taste. If User A and User B both completed 5 episodes of “Stranger Things”, “Dark”, and “Money Heist”, then we can reasonably infer that a title User A watched but User B hasn’t is a good recommendation for User B - even if we don’t know that all three watched titles happen to be tense, plot-driven thriller series.

The mathematical formulation is elegant. Think of it as describing a library with a seating arrangement that reflects reader compatibility. We don’t label seats “mystery readers” or “romance readers”; instead, we assign each reader (user) and each book (title) a position in a 128-dimensional latent space, and readers sitting close to books are predicted to enjoy those books. The key word is “latent” - these dimensions don’t correspond to human-interpretable concepts like “thriller” or “slow-burn”. They emerge from the data itself.

Matrix factorization decomposes the user-item interaction matrix R (280M users x 15K titles, almost entirely zeros) into two smaller matrices: P (users x 128 latent dimensions) and Q (items x 128 latent dimensions), such that R is approximately equal to P times Q transposed. The predicted score for user u and item i is the dot product of P[u] and Q[i].

# ALS Matrix Factorization for implicit feedback (watch fraction as confidence)
# Implements the Hu, Koren, Volinsky (2008) iALS approach used in production systems
import numpy as np
from scipy.sparse import csr_matrix

def als_step_user(
    R: csr_matrix,          # (n_users x n_items) sparse implicit rating matrix
    Q: np.ndarray,          # (n_items x k) current item factor matrix
    lmbda: float = 0.01,    # L2 regularization strength
    alpha: float = 40.0,    # confidence scaling: c_ui = 1 + alpha * r_ui
) -> np.ndarray:
    """
    Solve for all user factors P given fixed item factors Q.
    iALS: each user has a confidence-weighted least squares problem.
    Returns updated P matrix of shape (n_users x k).
    """
    n_users, k = R.shape[0], Q.shape[1]
    P = np.zeros((n_users, k))

    # Precompute Q^T Q (same for all users)
    QTQ = Q.T @ Q  # (k x k)
    reg_matrix = lmbda * np.eye(k)

    for u in range(n_users):
        # Get items this user interacted with
        user_items = R.getrow(u)
        item_indices = user_items.nonzero()[1]

        if len(item_indices) == 0:
            # New user: initialize with zero vector
            P[u] = np.zeros(k)
            continue

        # Build confidence-weighted system for this user
        r_u = np.array(user_items[0, item_indices]).flatten()
        c_u = 1.0 + alpha * r_u  # confidence weights

        # Q_u: rows of Q for items this user interacted with
        Q_u = Q[item_indices]  # (|I_u| x k)

        # Confidence-weighted QTQ for this user's observed items
        # Full iALS: A = Q^T (C_u - I) Q + QTQ + lambda*I
        # Simplified for sparse: only iterate over observed items
        CQ_u = Q_u * (c_u - 1.0)[:, np.newaxis]   # scale rows by (c_ui - 1)
        A = QTQ + Q_u.T @ CQ_u + reg_matrix  # (k x k)

        # b = Q^T C_u p_u (where preference p_ui = 1 for all observed)
        b = (Q_u * c_u[:, np.newaxis]).sum(axis=0)  # (k,)

        # Solve A * p_u = b (linear system, k is small: 128)
        P[u] = np.linalg.solve(A, b)

    return P


def als_step_item(
    R: csr_matrix,
    P: np.ndarray,
    lmbda: float = 0.01,
    alpha: float = 40.0,
) -> np.ndarray:
    """
    Solve for all item factors Q given fixed user factors P.
    Symmetric to als_step_user.
    """
    return als_step_user(R.T.tocsr(), P, lmbda, alpha)


def train_als(
    R: csr_matrix,
    k: int = 128,
    n_epochs: int = 20,
    lmbda: float = 0.01,
    alpha: float = 40.0,
) -> tuple[np.ndarray, np.ndarray]:
    """
    Alternating Least Squares training loop.
    Returns (P, Q) - user and item factor matrices.
    """
    n_users, n_items = R.shape
    # Initialize with small random values
    P = np.random.normal(0, 0.01, (n_users, k))
    Q = np.random.normal(0, 0.01, (n_items, k))

    for epoch in range(n_epochs):
        P = als_step_user(R, Q, lmbda, alpha)
        Q = als_step_item(R, P, lmbda, alpha)

        if epoch % 5 == 0:
            # Compute training loss (RMSE on observed entries) for monitoring
            observed = R.nonzero()
            pred = (P[observed[0]] * Q[observed[1]]).sum(axis=1)
            rmse = np.sqrt(((pred - np.ones(len(pred)))**2).mean())
            print(f"Epoch {epoch}: RMSE on observed = {rmse:.4f}")

    return P, Q

The critical detail in the above is the confidence weighting from the Hu-Koren-Volinsky iALS paper. Rather than treating the 0s in the matrix as “user dislikes this title” (they don’t - the user may simply have never seen the title), the iALS formulation treats observed interactions as high-confidence signal (c_ui = 1 + 40 * r_ui) and missing entries as low-confidence “no preference” (c_ui = 1). A user who watched 90% of a 10-episode season has confidence weight 37 for each episode. A user who watched 5% of episode 1 before giving up has confidence 3. This distinction is what makes implicit feedback training converge to meaningful recommendations rather than simply predicting zero for everything.

Content-Based Features and Two-Tower Neural Networks

Collaborative filtering alone cannot solve the cold start problem. A title released today has zero historical interactions, so it cannot appear in anyone’s collaborative filtering recommendations until it accumulates enough viewing data - typically 100+ interactions. For a major new release, that delay is unacceptable.

Content-based filtering solves this by using the title’s own attributes - genre, director, cast, synopsis keywords, language, release year - to compute a similarity to other titles that the user has watched. If a user watches Korean thrillers, a new Korean thriller release can appear in their recommendations based purely on attribute similarity, before any collaborative signal exists.

Netflix implements this via a Two-Tower Neural Network that learns separate embedding functions for users and items. The user tower takes as input: the user’s watch history (as a bag of item embeddings), demographic signals (inferred, not explicit), device type, time of day, and recent context (last 3 titles watched). The item tower takes: genre vector (multi-hot), cast embeddings (averaged), synopsis TF-IDF vector, language, release year, and content tags. Both towers output a 256-dimensional embedding. The training objective is to maximize the cosine similarity between a user’s tower output and the item they watched, while minimizing similarity to randomly sampled items they did not watch.

# Two-Tower Neural Network for recommendation - PyTorch implementation
import torch
import torch.nn as nn
import torch.nn.functional as F

class UserTower(nn.Module):
    def __init__(
        self,
        item_emb_dim: int = 128,     # dimension of pre-trained item embeddings
        context_dim: int = 32,        # device + time-of-day features
        history_len: int = 50,        # last N watched items in history
        output_dim: int = 256,
    ):
        super().__init__()
        # Encode watch history via average pooling of item embeddings
        self.history_encoder = nn.Sequential(
            nn.Linear(item_emb_dim, 128),
            nn.ReLU(),
            nn.Linear(128, 128),
        )
        # Context encoder (device type, time bucket, day of week)
        self.context_encoder = nn.Sequential(
            nn.Linear(context_dim, 64),
            nn.ReLU(),
        )
        # Fusion MLP
        self.fusion = nn.Sequential(
            nn.Linear(128 + 64, 256),
            nn.ReLU(),
            nn.Linear(256, output_dim),
        )

    def forward(
        self,
        history_embeddings: torch.Tensor,  # (batch, history_len, item_emb_dim)
        history_mask: torch.Tensor,         # (batch, history_len) - 1 for real items
        context_features: torch.Tensor,     # (batch, context_dim)
    ) -> torch.Tensor:
        # Average pool over history (masked to exclude padding)
        mask = history_mask.unsqueeze(-1).float()  # (batch, history_len, 1)
        sum_emb = (history_embeddings * mask).sum(dim=1)   # (batch, item_emb_dim)
        count = mask.sum(dim=1).clamp(min=1)               # (batch, 1)
        avg_history = sum_emb / count                      # (batch, item_emb_dim)

        history_enc = self.history_encoder(avg_history)    # (batch, 128)
        context_enc = self.context_encoder(context_features)  # (batch, 64)

        combined = torch.cat([history_enc, context_enc], dim=-1)
        user_embedding = self.fusion(combined)             # (batch, output_dim)
        return F.normalize(user_embedding, dim=-1)         # unit norm for cosine similarity


class ItemTower(nn.Module):
    def __init__(
        self,
        genre_dim: int = 30,       # multi-hot genre vector size
        cast_emb_dim: int = 64,    # averaged cast member embeddings
        text_dim: int = 512,       # synopsis TF-IDF or text embedding
        output_dim: int = 256,
    ):
        super().__init__()
        self.genre_encoder = nn.Linear(genre_dim, 64)
        self.cast_encoder = nn.Linear(cast_emb_dim, 64)
        self.text_encoder = nn.Sequential(
            nn.Linear(text_dim, 256),
            nn.ReLU(),
            nn.Linear(256, 128),
        )
        self.fusion = nn.Sequential(
            nn.Linear(64 + 64 + 128, 256),
            nn.ReLU(),
            nn.Linear(256, output_dim),
        )

    def forward(
        self,
        genre_features: torch.Tensor,  # (batch, genre_dim)
        cast_embeddings: torch.Tensor, # (batch, cast_emb_dim)
        text_features: torch.Tensor,   # (batch, text_dim)
    ) -> torch.Tensor:
        genre_enc = F.relu(self.genre_encoder(genre_features))
        cast_enc = F.relu(self.cast_encoder(cast_embeddings))
        text_enc = self.text_encoder(text_features)

        combined = torch.cat([genre_enc, cast_enc, text_enc], dim=-1)
        item_embedding = self.fusion(combined)
        return F.normalize(item_embedding, dim=-1)


def two_tower_loss(
    user_emb: torch.Tensor,   # (batch, dim)
    pos_item_emb: torch.Tensor,  # (batch, dim) - item user watched
    neg_item_emb: torch.Tensor,  # (batch, neg_samples, dim) - items user did not watch
    temperature: float = 0.07,
) -> torch.Tensor:
    """
    In-batch negative sampling loss (sampled softmax).
    pos_score should be high; neg_scores should be low.
    """
    # Positive scores
    pos_scores = (user_emb * pos_item_emb).sum(dim=-1) / temperature  # (batch,)

    # Negative scores: user vs. all negatives
    # user_emb: (batch, dim), neg_item_emb: (batch, neg_k, dim)
    neg_scores = torch.bmm(neg_item_emb, user_emb.unsqueeze(-1)).squeeze(-1) / temperature  # (batch, neg_k)

    # Softmax loss: maximize probability of positive over all candidates
    all_scores = torch.cat([pos_scores.unsqueeze(1), neg_scores], dim=1)  # (batch, 1+neg_k)
    labels = torch.zeros(user_emb.shape[0], dtype=torch.long, device=user_emb.device)
    loss = F.cross_entropy(all_scores, labels)
    return loss

The Two-Tower model handles cold start because the item tower uses only content features - it can generate an embedding for a brand-new title the moment its metadata is available (before any user has watched it). The user tower uses their historical item embeddings as input, so it also works for new users as soon as they watch their first title.

The Feature Store

The Feature Store is the central nervous system connecting offline training and online serving. It stores two categories of features: item features (static attributes like genre, cast, language - rarely change), and user features (dynamic attributes like recent watch history, inferred preferences, viewing velocity - change hourly).

A critical distinction: item features are used by both the Training Pipeline (to build Two-Tower inputs) and the Serving Layer (to populate the ranking model’s input). User features are used similarly. Without a shared store, you get training-serving skew - the features used to train the model differ from the features used to serve predictions, causing the model’s offline accuracy to not translate to online performance.

-- Feature Store schema: user feature snapshot updated hourly
CREATE TABLE user_feature_snapshot (
    user_id             BIGINT          NOT NULL,
    snapshot_hour       TIMESTAMPTZ     NOT NULL,  -- truncated to hour
    watch_history_ids   BIGINT[]        NOT NULL,  -- last 50 title_ids watched
    preferred_genres    FLOAT[],        -- 30-dim soft preference vector
    avg_session_length_m FLOAT,         -- average session duration in minutes
    days_since_last_watch INT,
    preferred_device    VARCHAR(16),    -- mobile, tv, desktop
    preferred_watch_hour SMALLINT,      -- 0-23, most common viewing hour
    country_code        CHAR(2)         NOT NULL,
    language_preference VARCHAR(8),
    content_maturity_max VARCHAR(8),    -- G, PG, R, NC-17, etc.
    watch_velocity_7d   FLOAT,          -- titles fully watched in last 7 days
    PRIMARY KEY (user_id, snapshot_hour)
) PARTITION BY RANGE (snapshot_hour);

-- Item feature table: updated when catalog changes
CREATE TABLE item_feature (
    title_id            BIGINT          PRIMARY KEY,
    title_name          VARCHAR(256)    NOT NULL,
    release_year        SMALLINT,
    duration_minutes    INTEGER,
    content_type        VARCHAR(16)     NOT NULL,  -- movie, series, short
    maturity_rating     VARCHAR(8),
    primary_language    VARCHAR(8),
    genre_vector        FLOAT[],        -- 30-dim multi-hot, normalized
    avg_cast_embedding  FLOAT[],        -- 64-dim average of top-5 cast embeddings
    synopsis_embedding  FLOAT[],        -- 512-dim text embedding of synopsis
    available_countries CHAR(2)[],
    available_since     TIMESTAMPTZ     NOT NULL,
    total_view_count    BIGINT          NOT NULL DEFAULT 0,
    avg_completion_pct  FLOAT,
    updated_at          TIMESTAMPTZ     NOT NULL DEFAULT NOW()
);

CREATE INDEX idx_item_country ON item_feature USING GIN (available_countries);
CREATE INDEX idx_item_language ON item_feature (primary_language);
CREATE INDEX idx_item_type ON item_feature (content_type, release_year DESC);

Candidate Generation and Ranking

The serving path has two stages: fast candidate generation (retrieve ~500 titles that might be relevant) and precise ranking (score all 500 with full context features to produce the final ranked list). This separation is the pattern every large recommendation system uses - it’s the same structure as Google Search’s recall-then-rank pipeline.

Candidate generation uses Approximate Nearest Neighbor search. The 128-dimensional user vector from the Embedding Store is used as a query against a FAISS (Facebook AI Similarity Search) HNSW index of all 15,000 item vectors. HNSW (Hierarchical Navigable Small World) is a graph-based ANN algorithm that achieves sub-millisecond query times on millions of items by building a multi-layer navigation structure. At query time, it traverses from coarse-to-fine layers, converging on the approximate nearest neighbors without scanning all 15,000 vectors.

Multiple candidate generators run in parallel: the collaborative filtering ANN (captures “users like you watched this”), the content-based Two-Tower ANN (captures “based on what you watched, you’d like this”), a trending/popularity generator (captures “everyone is watching this right now”), and a “continue watching” generator (titles with incomplete watch history). The outputs are merged and deduplicated before passing to the Ranker.

# Candidate generation: ANN search + multi-source merge for homepage recommendations
import faiss
import numpy as np
from typing import List, Dict, Set

class CandidateGenerator:
    def __init__(
        self,
        item_vectors: np.ndarray,   # (n_items, dim) - ALS item embeddings
        item_ids: List[int],         # mapping from FAISS index row -> title_id
        dim: int = 128,
    ):
        # Build FAISS HNSW index (graph-based ANN, fast query, no retraining needed)
        self.index = faiss.IndexHNSWFlat(dim, 32)  # M=32 neighbors per layer
        self.index.hnsw.efConstruction = 200       # construction quality
        self.index.hnsw.efSearch = 64              # query-time quality tradeoff
        normalized = item_vectors / (np.linalg.norm(item_vectors, axis=1, keepdims=True) + 1e-9)
        self.index.add(normalized.astype('float32'))
        self.item_ids = item_ids
        self.id_to_idx = {item_id: idx for idx, item_id in enumerate(item_ids)}

    def retrieve_by_user_vector(
        self,
        user_vector: np.ndarray,  # (dim,)
        k: int = 500,
    ) -> List[tuple[int, float]]:
        """
        Returns list of (title_id, similarity_score) tuples, sorted by similarity.
        Uses inner product (cosine similarity for normalized vectors).
        """
        user_vec_norm = user_vector / (np.linalg.norm(user_vector) + 1e-9)
        query = user_vec_norm.reshape(1, -1).astype('float32')
        scores, indices = self.index.search(query, k)
        return [
            (self.item_ids[int(idx)], float(score))
            for idx, score in zip(indices[0], scores[0])
            if idx >= 0  # FAISS returns -1 for padded results
        ]

    def retrieve_by_item_similarity(
        self,
        seed_title_id: int,  # "because you watched X" source
        k: int = 200,
        exclude_ids: Set[int] = None,
    ) -> List[tuple[int, float]]:
        """
        Returns titles similar to the seed title (item-to-item collaborative filtering).
        Powers "Because You Watched" rows.
        """
        if seed_title_id not in self.id_to_idx:
            return []
        seed_idx = self.id_to_idx[seed_title_id]
        seed_vec = self.index.reconstruct(seed_idx).reshape(1, -1)
        scores, indices = self.index.search(seed_vec, k + 1)
        results = []
        for idx, score in zip(indices[0], scores[0]):
            if idx < 0 or int(idx) == seed_idx:
                continue
            title_id = self.item_ids[int(idx)]
            if exclude_ids and title_id in exclude_ids:
                continue
            results.append((title_id, float(score)))
            if len(results) >= k:
                break
        return results


def merge_candidates(
    collab_candidates: List[tuple[int, float]],      # (title_id, score)
    content_candidates: List[tuple[int, float]],
    trending_candidates: List[tuple[int, float]],
    continue_watching: List[int],                     # title_ids with incomplete watch
    watched_ids: Set[int],                            # exclude already completed
    max_candidates: int = 500,
) -> List[tuple[int, float, str]]:
    """
    Merges candidates from multiple sources, deduplicates, and returns top-N.
    Returns (title_id, score, source) tuples.
    """
    seen = set(watched_ids)
    merged: Dict[int, tuple[float, str]] = {}

    # Priority: continue watching first (user already started)
    for title_id in continue_watching:
        if title_id not in seen:
            merged[title_id] = (2.0, "continue_watching")  # boost score
            seen.add(title_id)

    # Collaborative filtering candidates
    for title_id, score in collab_candidates:
        if title_id not in seen and title_id not in merged:
            merged[title_id] = (score * 1.0, "collab")

    # Content-based candidates (slightly lower weight to prefer collab)
    for title_id, score in content_candidates:
        if title_id not in seen and title_id not in merged:
            merged[title_id] = (score * 0.9, "content")

    # Trending (ensures new releases have representation)
    for title_id, score in trending_candidates:
        if title_id not in seen and title_id not in merged:
            merged[title_id] = (score * 0.7, "trending")

    sorted_candidates = sorted(merged.items(), key=lambda x: x[1][0], reverse=True)
    return [(tid, score, src) for tid, (score, src) in sorted_candidates[:max_candidates]]

The Ranking stage takes the 500 merged candidates and re-scores them with an XGBoost gradient boosting model that incorporates features the ANN search cannot see: contextual signals (what time of day is it? what device is the user on? is this a weeknight or weekend?), interaction cross-features (user prefers action + item is action = high score; user prefers Korean + item is Korean = boost), and quality signals (item completion rate, item play rate by similar users). XGBoost produces a final score for each of the 500 candidates. The top 75 are passed to the post-ranking filter.

Contextual Signals and Model Freshness

A user’s preferences are not static. Someone who normally watches drama series will switch to comedy after a stressful week at work. Someone on a mobile phone during a commute wants a 30-minute episode, not a 2-hour movie. These contextual signals - device type, time of day, day of week, viewing session length history - are critical for ranking but are not captured in the offline-trained embedding models (which represent a user’s historical average preferences, not their current-moment intent).

The near-real-time layer addresses this by maintaining a session context store in Redis. When a user opens the app, the last 3 watched items from the past 24 hours are loaded from a Redis sorted set (sorted by watch timestamp). These recent items are passed directly to the ranking model as features, allowing the ranker to adjust its output based on what the user watched today - even if the offline model was trained on data from 6 hours ago.

# Near-real-time user session context: recent watches and computed features
import redis
import json
import time

class UserSessionContext:
    def __init__(self, redis_client: redis.Redis):
        self.redis = redis_client
        self.key_prefix = "user:session:"
        self.ttl_seconds = 86400  # 24 hour TTL

    def record_watch_event(
        self,
        user_id: int,
        title_id: int,
        watch_fraction: float,
        duration_minutes: int,
        device_type: str,
    ) -> None:
        """
        Called by Flink stream processor on each completed watch event.
        Maintains a sorted set of recent watches (score = timestamp).
        """
        key = f"{self.key_prefix}{user_id}:watches"
        event = json.dumps({
            "title_id": title_id,
            "watch_fraction": watch_fraction,
            "duration_m": duration_minutes,
            "device": device_type,
        })
        score = time.time()
        pipe = self.redis.pipeline()
        # Add to sorted set with timestamp as score
        pipe.zadd(key, {event: score})
        # Keep only last 50 events
        pipe.zremrangebyrank(key, 0, -51)
        pipe.expire(key, self.ttl_seconds)
        pipe.execute()

    def get_recent_context(
        self,
        user_id: int,
        n_recent: int = 10,
    ) -> list[dict]:
        """
        Returns N most recent watch events for this user (last 24h max).
        Used at serving time to populate contextual features for ranker.
        """
        key = f"{self.key_prefix}{user_id}:watches"
        # Get last n items by score (most recent)
        raw = self.redis.zrevrange(key, 0, n_recent - 1, withscores=False)
        events = []
        for item in raw:
            try:
                events.append(json.loads(item))
            except json.JSONDecodeError:
                continue
        return events

    def get_session_features(self, user_id: int) -> dict:
        """
        Derives features from recent context for use in ranking model.
        """
        context = self.get_recent_context(user_id, n_recent=20)
        if not context:
            return {
                "recent_title_ids": [],
                "recent_avg_completion": 0.5,
                "recent_dominant_device": "unknown",
                "recent_avg_duration_m": 45.0,
                "last_watch_hours_ago": 48.0,
            }

        completions = [e["watch_fraction"] for e in context]
        durations = [e["duration_m"] for e in context]
        devices = [e["device"] for e in context]
        recent_ids = [e["title_id"] for e in context[:5]]

        return {
            "recent_title_ids": recent_ids,
            "recent_avg_completion": sum(completions) / len(completions),
            "recent_dominant_device": max(set(devices), key=devices.count),
            "recent_avg_duration_m": sum(durations) / len(durations),
            "last_watch_hours_ago": (time.time() - float(
                self.redis.zscore(
                    f"{self.key_prefix}{user_id}:watches",
                    self.redis.zrange(f"{self.key_prefix}{user_id}:watches", -1, -1)[0]
                ) or time.time() - 86400
            )) / 3600,
        }

Model freshness is the gap between when an interaction happens and when it influences recommendations. The target is 1-6 hours. This is achieved through a two-level update cadence:

1. Near-real-time profile updates (under 1 hour): The Redis session context captures new watches immediately. The Flink pipeline also updates lightweight preference vectors (30-dimensional genre preference scores, updated with exponential decay) in Redis, incorporating the new watch within minutes.

2. Full model retraining (every 1-6 hours): The ALS/Two-Tower models are retrained on a rolling 7-day window of all interaction data. New user and item vectors are published to the embedding store. The FAISS ANN index on serving nodes is hot-swapped with the new item vectors (a process that takes a few seconds using FAISS’s in-memory swap mechanism).

The Cold Start Problem

Cold start is one of the hardest problems in recommendation systems. It has two forms: user cold start (a brand-new subscriber with zero history) and item cold start (a brand-new title with zero viewing data).

For user cold start, Netflix employs a combination of onboarding signals and popularity-based fallback. During signup, the user is asked to select 3+ titles they enjoy (an explicit genre/title signal). From these selections, a proxy user vector is constructed by averaging the item vectors of the selected titles. This proxy vector is good enough to seed the collaborative filtering candidate retrieval. For users who skip onboarding, Netflix falls back to country-and-language-specific popularity rankings for the first homepage, relying on the first few real watches to rapidly converge to a personalized vector.

For item cold start, the Two-Tower model’s item tower generates a content-based embedding immediately from catalog metadata (genre, cast, synopsis). This embedding is added to the FAISS ANN index as soon as the item is available for streaming. Users whose collaborative filtering vectors land near this content embedding in the latent space will see the new title in their candidate set from day one, even before a single user has watched it.

# Cold start: construct proxy user vector from onboarding title selections
import numpy as np
from typing import List

def cold_start_user_vector(
    selected_title_ids: List[int],
    item_vectors: dict[int, np.ndarray],   # title_id -> 128-dim ALS vector
    content_embeddings: dict[int, np.ndarray],  # title_id -> 256-dim two-tower vector
    country_popular_ids: List[int],        # top-100 titles popular in user's country
    dim: int = 128,
) -> np.ndarray:
    """
    Constructs an initial user vector for a new user from their onboarding selections.
    Returns a 128-dim vector suitable for ANN candidate retrieval.
    """
    if not selected_title_ids:
        # Zero-history fallback: use country popularity centroid
        pop_vecs = [item_vectors[tid] for tid in country_popular_ids[:20] if tid in item_vectors]
        if pop_vecs:
            return np.mean(pop_vecs, axis=0)
        return np.zeros(dim)

    # Average item vectors for selected titles
    selected_vecs = []
    for title_id in selected_title_ids:
        if title_id in item_vectors:
            selected_vecs.append(item_vectors[title_id])

    if not selected_vecs:
        return np.zeros(dim)

    user_vec = np.mean(selected_vecs, axis=0)

    # Normalize
    norm = np.linalg.norm(user_vec)
    if norm > 1e-9:
        user_vec = user_vec / norm

    return user_vec


def cold_start_item_embedding(
    title_metadata: dict,          # genre_vector, cast_embedding, synopsis_embedding
    item_tower: 'ItemTower',       # pre-trained Two-Tower item model
) -> np.ndarray:
    """
    Generates an item embedding for a brand-new title using only content features.
    Called immediately when a new title is made available for streaming.
    """
    import torch
    genre = torch.tensor([title_metadata['genre_vector']], dtype=torch.float32)
    cast = torch.tensor([title_metadata['cast_embedding']], dtype=torch.float32)
    text = torch.tensor([title_metadata['synopsis_embedding']], dtype=torch.float32)

    with torch.no_grad():
        embedding = item_tower(genre, cast, text)

    return embedding.numpy()[0]

A/B Testing Framework and Model Rollout

The A/B Testing Framework is what allows Netflix to safely deploy new recommendation models to a fraction of users, measure impact on real engagement metrics, and promote or roll back without human intervention. Think of it as a continuous scientific experiment running on 280 million subjects, with automated decisions on which experimental treatments to scale.

Every user is assigned to exactly one experiment bucket, determined by hash(user_id + experiment_id) % 1000. A user in bucket 0-499 gets the control model; buckets 500-699 get experiment A; buckets 700-849 get experiment B. Assignments are deterministic and stable across sessions - the same user always gets the same model, enabling coherent before/after comparisons.

The key metrics are play rate (% of recommended tiles that result in a start), completion rate (% of started titles watched to at least 80%), and 28-day retention (are users still subscribed). Play rate is the fastest-to-measure signal (hours) and serves as the primary decision metric. Completion rate takes days. Retention takes months. Netflix’s A/B framework makes promotion decisions based on play rate and completion rate with a minimum two-week observation window (to avoid false positives from novelty effects: a new recommendation UI always gets clicks initially just because it’s different).

# A/B test significance evaluation: two-proportion z-test for play rates
import math
from dataclasses import dataclass
from typing import Optional

@dataclass
class ExperimentResult:
    control_plays: int
    control_impressions: int
    treatment_plays: int
    treatment_impressions: int

def two_proportion_z_test(result: ExperimentResult) -> tuple[float, float]:
    """
    Tests whether treatment play rate is significantly different from control.
    Returns (z_score, p_value). p < 0.05 = statistically significant.
    """
    p1 = result.control_plays / result.control_impressions
    p2 = result.treatment_plays / result.treatment_impressions
    n1 = result.control_impressions
    n2 = result.treatment_impressions

    # Pooled proportion under null hypothesis
    p_pool = (result.control_plays + result.treatment_plays) / (n1 + n2)

    if p_pool * (1 - p_pool) < 1e-10:
        return 0.0, 1.0  # degenerate case

    se = math.sqrt(p_pool * (1 - p_pool) * (1/n1 + 1/n2))
    if se < 1e-10:
        return 0.0, 1.0

    z = (p2 - p1) / se

    # Two-tailed p-value approximation (using error function)
    import math
    p_value = 2 * (1 - 0.5 * (1 + math.erf(abs(z) / math.sqrt(2))))
    return z, p_value

def should_promote(
    result: ExperimentResult,
    min_lift_pct: float = 1.0,    # require at least 1% relative improvement
    max_p_value: float = 0.05,    # 95% confidence threshold
    min_impressions: int = 100_000,  # minimum statistical power
) -> tuple[bool, str]:
    """
    Automated promotion decision: returns (should_promote, reason).
    """
    if result.treatment_impressions < min_impressions:
        return False, f"Insufficient data: {result.treatment_impressions} < {min_impressions}"

    p1 = result.control_plays / result.control_impressions
    p2 = result.treatment_plays / result.treatment_impressions
    lift_pct = (p2 - p1) / p1 * 100

    if lift_pct < min_lift_pct:
        return False, f"Lift {lift_pct:.2f}% below threshold {min_lift_pct}%"

    z, p_value = two_proportion_z_test(result)

    if p_value > max_p_value:
        return False, f"Not significant: p={p_value:.4f} > {max_p_value}"

    return True, f"Promote: lift={lift_pct:.2f}%, z={z:.2f}, p={p_value:.4f}"

Data Model

-- Core recommendation engine data model

-- User embedding store: primary source for candidate retrieval
CREATE TABLE user_embedding (
    user_id             BIGINT          NOT NULL,
    model_version       VARCHAR(32)     NOT NULL,  -- e.g., "als_v42"
    embedding_vector    FLOAT[]         NOT NULL,  -- 128-dim latent factor
    embedding_norm      FLOAT           NOT NULL,
    trained_at          TIMESTAMPTZ     NOT NULL,
    training_event_count BIGINT         NOT NULL,  -- events used in this training run
    PRIMARY KEY (user_id, model_version)
);
CREATE INDEX idx_user_emb_model ON user_embedding (model_version, trained_at DESC);

-- Item embedding store: indexed in FAISS on serving nodes
CREATE TABLE item_embedding (
    title_id            BIGINT          NOT NULL,
    model_version       VARCHAR(32)     NOT NULL,
    embedding_vector    FLOAT[]         NOT NULL,  -- 128-dim for ALS, 256-dim for Two-Tower
    embedding_type      VARCHAR(16)     NOT NULL CHECK (embedding_type IN ('collab', 'content', 'two_tower')),
    trained_at          TIMESTAMPTZ     NOT NULL,
    PRIMARY KEY (title_id, model_version, embedding_type)
);

-- A/B experiment configuration
CREATE TABLE ab_experiment (
    experiment_id       VARCHAR(64)     PRIMARY KEY,
    experiment_name     VARCHAR(128)    NOT NULL,
    layer               VARCHAR(32)     NOT NULL,  -- candidate, ranker, ui, post_filter
    start_time          TIMESTAMPTZ     NOT NULL,
    end_time            TIMESTAMPTZ,
    status              VARCHAR(16)     NOT NULL DEFAULT 'running'
        CHECK (status IN ('running', 'promoted', 'rolled_back', 'completed')),
    control_traffic_pct FLOAT           NOT NULL DEFAULT 50.0,
    treatment_config    JSONB           NOT NULL,  -- model config for treatment arm
    promotion_criteria  JSONB           NOT NULL   -- min_lift_pct, max_p_value, min_impressions
);

-- Recommendation impression log: training data for next model iteration
CREATE TABLE recommendation_impression (
    impression_id       BIGINT          GENERATED ALWAYS AS IDENTITY,
    user_id             BIGINT          NOT NULL,
    title_id            BIGINT          NOT NULL,
    experiment_id       VARCHAR(64),
    treatment_arm       VARCHAR(16),    -- control, treatment_a, treatment_b
    rank_position       SMALLINT        NOT NULL,  -- 1-indexed position on homepage
    row_type            VARCHAR(32)     NOT NULL,  -- top_picks, because_you_watched, trending
    retrieval_score     FLOAT,
    ranking_score       FLOAT,
    was_clicked         BOOLEAN         NOT NULL DEFAULT FALSE,
    was_played          BOOLEAN         NOT NULL DEFAULT FALSE,
    play_fraction       FLOAT,          -- 0-1, filled in after play completes
    served_at           TIMESTAMPTZ     NOT NULL DEFAULT NOW()
)
PARTITION BY RANGE (served_at);
-- Daily partitions, 90-day retention for training data

CREATE INDEX idx_impression_user ON recommendation_impression (user_id, served_at DESC);
CREATE INDEX idx_impression_exp ON recommendation_impression (experiment_id, served_at DESC)
    WHERE experiment_id IS NOT NULL;

Key Algorithms and Protocols

Approximate Nearest Neighbor with HNSW

FAISS HNSW (Hierarchical Navigable Small World) is the core algorithm enabling sub-millisecond candidate retrieval across 15,000 item vectors. Think of it as a highway network layered on top of a city street map. The top layer connects only major “landmark” nodes that are well-separated - this is the highway. Descending to lower layers adds more nodes (smaller roads). A query starts at the top layer, greedily moves toward the nearest node, drops to the next layer, and continues until reaching the bottom layer where all items exist.

# FAISS HNSW index build and query with hot-swap for model updates
import faiss
import numpy as np
import threading

class RecommendationIndex:
    def __init__(self, dim: int = 128, ef_construction: int = 200, M: int = 32):
        """
        M: number of bi-directional links per node (higher = better recall, more memory)
        ef_construction: construction beam width (higher = better index quality, slower build)
        """
        self.dim = dim
        self.M = M
        self.ef_construction = ef_construction
        self._lock = threading.RLock()
        self._index = None
        self._item_ids = []

    def build(self, item_vectors: np.ndarray, item_ids: list[int], ef_search: int = 64) -> None:
        """
        Builds a new HNSW index from item vectors.
        Called when a new model version is published (hourly).
        """
        index = faiss.IndexHNSWFlat(self.dim, self.M)
        index.hnsw.efConstruction = self.ef_construction
        index.hnsw.efSearch = ef_search

        # Normalize for cosine similarity
        vecs = item_vectors.astype('float32')
        norms = np.linalg.norm(vecs, axis=1, keepdims=True)
        vecs = vecs / np.maximum(norms, 1e-9)

        index.add(vecs)

        # Atomic hot-swap: replace old index without dropping queries in flight
        with self._lock:
            self._index = index
            self._item_ids = list(item_ids)

    def query(
        self,
        user_vector: np.ndarray,  # (dim,)
        k: int = 500,
    ) -> list[tuple[int, float]]:
        """
        Finds k approximate nearest neighbors to user_vector.
        Thread-safe: holds read lock during index access.
        """
        with self._lock:
            if self._index is None:
                return []
            index = self._index
            ids = self._item_ids

        user_vec = user_vector.astype('float32').reshape(1, -1)
        norm = np.linalg.norm(user_vec)
        if norm > 1e-9:
            user_vec = user_vec / norm

        scores, indices = index.search(user_vec, k)
        return [
            (ids[int(idx)], float(score))
            for idx, score in zip(indices[0], scores[0])
            if idx >= 0
        ]

The query time complexity of HNSW is O(log N) for N items - for N=15,000 this is essentially constant. The index size is approximately 4 * dim * M * N bytes = 4 * 128 * 32 * 15000 = ~246 MB - trivially fits in memory on a serving node.

Scaling and Performance

Capacity Estimation - Netflix Recommendation Engine

Given:
  - 280 million users
  - Peak 30M concurrent homepage loads
  - 300,000 homepage requests/second at peak
  - 15,000 items in catalog
  - 128-dim user + item embeddings

Embedding Store size:
  - User embeddings: 280M * 128 * 4 bytes (float32) = 143 GB
  - Item embeddings: 15K * 128 * 4 bytes = 7.5 MB (fits in L3 cache on any server)
  - User embeddings sharded across 10 Cassandra nodes: ~14 GB per node
  - Redis cache: top 5M active users * 512 bytes per embedding = 2.5 GB
  - Cache hit rate target: >95% (the active 5M users are making requests right now)

ANN Query load:
  - 300K req/s * 1 FAISS query per request = 300K ANN queries/second
  - Each query: ~0.5ms on a single CPU core
  - Single core can handle ~2000 queries/sec
  - Serving nodes needed: 300K / 2000 = 150 CPU cores for ANN
  - With 32-core servers: ~5 serving nodes minimum; use 20 for redundancy

Ranker (XGBoost) load:
  - 300K req/s * 500 candidates per request = 150M scores/second
  - XGBoost tree inference: ~10 microseconds per candidate
  - Single core: ~100K candidates/second
  - Total cores for ranking: 150M / 100K = 1500 cores
  - 30 servers of 64 cores each (with headroom)

Training data volume:
  - 10 billion interaction events per day
  - 7-day window = 70 billion events
  - At 200 bytes per event (user_id, title_id, watch_fraction, timestamp): 14 TB
  - S3 storage for 7-day training window: ~14 TB
  - ALS training on 14 TB at 1 Gbps shuffle bandwidth: ~30 minutes on 100 Spark nodes

Model publishing:
  - User embedding matrix: 143 GB written to Cassandra on each retrain
  - At 1 Gbps write bandwidth: ~20 minutes to publish
  - Item embedding matrix: 7.5 MB - immediate (<1 second)
  - FAISS index rebuild: ~30 seconds for 15K items on 1 CPU

Failure Modes and Recovery

Failure	Detection	Impact	Recovery
User embedding store unavailable (Cassandra nodes down)	Health check fails within 15s; serving latency spikes	User vector lookup returns null; serving falls back to popularity-based recommendations	Cassandra auto-recovery from replicas (replication factor 3); Redis cache serves stale embeddings for up to 1 hour if Cassandra is recovering
ALS training job failure (OOM or Spark node crash)	Job fails with non-zero exit code; monitoring detects embedding freshness > 8 hours	Recommendations become stale; users watching new content don’t see recency signal	Automatic job retry with increased memory; fallback to last successful model; alert on-call if freshness > 12 hours
FAISS index corruption (NaN in vectors, index rebuild failure)	Serving latency spike + zero-result queries	All users receive empty candidate sets; fallback to popularity-only recommendations	Serving nodes detect NaN results and fall back to content-based (Two-Tower) index which is rebuilt independently; FAISS rebuild from clean vectors restores full service
Feature Store Redis cache miss storm (cold restart)	Cache hit rate drops below 50%; latency spikes	Each request reads from Cassandra directly; 5-10x latency increase	Cassandra handles increased load (designed for 5x normal throughput); Redis warms back up over 10-15 minutes as requests refill the cache
A/B experiment bucketing misconfiguration (overlapping experiments in same layer)	p-values too low, control group behavior changes unexpectedly	Invalid experiment results; may cause poor experience for users in conflicting buckets	Experiment service validates no overlapping traffic allocations at creation time; conflicting experiments auto-paused with alert
Training-serving skew (feature preprocessing differs between train and serve)	Offline metrics look good, but online play rate drops after model deployment	Model underperforms expectations; not obvious which feature is mismatched	Shadow traffic testing (running new model offline against live requests before deployment) catches skew before it affects users; feature importance tracking identifies the mismatched feature

Comparison of Approaches

Approach	Accuracy	Cold Start	Freshness	Compute Cost	Best Fit
Collaborative Filtering (ALS)	High (for active users)	Fails (no history)	Hours (batch retrain)	Medium (batch Spark)	Active users, stable catalog
Content-Based Filtering	Medium	Excellent (only needs metadata)	Immediate (metadata-driven)	Low	New users, new items
Two-Tower Neural Network	Very High	Good (item tower uses metadata)	Hours (model retrain)	High (GPU training)	Combined user + item signal
Session-based (RNN/Transformer)	Very High for short-term	Excellent	Sub-minute	Very High	In-session “next item” recommendations
Popularity-Based	Low (same for all)	Excellent	Minutes	Minimal	Cold start fallback, trending rows
Hybrid (current system)	Highest	Good	1-6 hours	High	Production at scale

The hybrid approach wins in production because no single algorithm handles all cases well. Collaborative filtering is best for active users with substantial history. Content-based handles new users and new items. Two-Tower handles both but requires more infrastructure. Popularity ensures the “Trending Now” row always has fresh content. The key is building a merging and ranking layer that can blend candidates from all sources and let the ranking model decide the final score.

The non-obvious trade-off is between freshness and accuracy. A model retrained every 5 minutes would have excellent freshness but would be trained on very little new data (just the last 5 minutes of interactions) - accuracy would suffer. A model retrained weekly has excellent accuracy (trained on 70 billion events) but completely misses this week’s trending content. The 1-6 hour window with a near-real-time context layer is the empirically correct balance point for a 280-million-user platform.

Key Takeaways

• Matrix factorization (ALS) on implicit feedback converts raw watch events into 128-dimensional latent space representations where proximity between a user vector and an item vector predicts interest - without ever manually defining what the latent dimensions mean.
• Two-stage candidate + rank architecture is mandatory at scale: ANN retrieval finds 500 candidates in 5ms; the ranking model scores them with full context in 10ms. Doing either stage alone is either too slow or too imprecise.
• Cold start requires hybrid signals: new users get a proxy vector from their onboarding selections; new titles get a Two-Tower content embedding from metadata alone - no viewing data required for either.
• Near-real-time vs. offline freshness are solved differently: Redis session context captures last-24-hour viewing recency cheaply; full ALS retrains every 1-6 hours capture deep signal that recency alone cannot. Both are necessary.
• Training-serving skew is the most dangerous failure mode in ML recommendation systems and is prevented by ensuring features are computed using the same code path at train and serve time.
• A/B testing in isolated layers prevents interaction effects: candidate generation experiments and ranking experiments run in independent layers so their results are statistically independent.
• Context signals (device, time-of-day, day-of-week) add 10-15% relative improvement in play rate over pure collaborative filtering - the same user has different content preferences in different situations.
• HNSW ANN index achieves 99%+ recall at 500 neighbors for a 15,000-item catalog in under 1ms, enabling the serving latency SLA without scanning all items.

The deepest counter-intuitive lesson from this system is that the user’s “taste” as captured by the ALS vector is not stable - it is a snapshot of preferences averaged over a historical window. The freshness challenge is not just about capturing new titles; it is about capturing changes in a user’s current-moment preferences that diverge from their historical average. Someone who binge-watched a true crime series last weekend is not necessarily a “true crime person” - they may have been in a specific mood that won’t persist. Treating collaborative filtering as a permanent taste profile, rather than a sliding window snapshot, causes recommendations to lag the user’s actual current preferences by days.

Frequently Asked Questions

Q: Why not use a deep learning model (like a Transformer) for the entire recommendation pipeline instead of ALS? A: Transformers achieve excellent recommendation quality but come with steep inference costs - a 12-layer Transformer scoring 500 candidates takes 100-200ms on a GPU, which blows the 100ms total SLA budget. For the candidate retrieval stage, ALS-based ANN search is 100x faster than Transformer-based retrieval. For the ranking stage, XGBoost is 10-20x faster than a small Transformer. Netflix does use session-based Transformer models for “continue watching” and “what to watch next” use cases where the user is already in an active session and latency budgets are more relaxed. The rule of thumb: use Transformers where you have one clear “next item” to predict; use matrix factorization + XGBoost where you need to generate a diverse list of 40+ titles.

Q: Why not build a separate recommendation model per country, given that content preferences vary significantly by region? A: A per-country model for 190+ countries would require 190 separate training runs, 190 separate FAISS indexes, and 190 separate serving paths - 190x the infrastructure cost and operational complexity. The better approach is a single global model with country as an explicit feature, and country-specific cold start fallbacks. The model learns that country correlates with genre preferences during training without needing to be completely retrained per country. For markets with very distinctive preferences (South Korea, India, Brazil), Netflix does deploy lightweight country-specific re-rankers as a final step - these are cheap boosting models that adjust the global model’s output, not full replacements.

Q: How do you prevent the recommendation engine from getting stuck in a filter bubble - always recommending the same genres the user has watched before? A: Diversity injection at the post-ranking stage explicitly ensures that the final 40 tiles span at least 4 different genres. The ranking model includes an “intra-list diversity” penalty that reduces the score of a candidate if 3+ similar items already appear in the current ranked list. Additionally, the “Trending Now” and “New Releases” rows bypass personalization entirely and use global popularity, ensuring every homepage contains some variety that the user’s personal model would not have surfaced. Netflix also occasionally injects “exploration” titles - items just outside the user’s current preference cluster - to gather feedback signals and prevent the model from over-specializing.

Q: Why does Netflix use 128 dimensions for the latent embedding rather than, say, 512 or 16? A: 128 dimensions is an empirical sweet spot for this catalog size. Fewer dimensions (16-32) lose expressiveness - the model cannot capture enough distinct taste factors to differentiate users who all watch “broadly popular” content but have different sub-preferences. More dimensions (512+) increase the ANN index memory footprint (the item matrix goes from 7.5 MB to 30 MB for 512-dim), slow down dot product computation, and can cause overfitting when the number of items is small relative to the dimension (15,000 items with 512 dimensions is underconstrained). The 128-dimension choice also fits neatly in 4 AVX SIMD instructions for fast vectorized dot product computation.

Q: How would you design the recommendation engine to handle a major new release that you expect to have 10 million views on its first day? A: Before the release: pre-compute content-based Two-Tower embeddings from the title metadata and inject them into all serving nodes’ FAISS indexes. Generate “teaser” recommendation candidates for users whose collaborative filtering vectors are close to the content embedding - these users see the title in their recommendations before anyone has watched it. On release day: monitor the title’s interaction count in near-real-time. Once it crosses 10,000 views (typically within an hour of release), its collaborative filtering signal is sufficient to meaningfully influence the ALS model. Trigger an out-of-schedule partial model update (retrain the item vector only, not the full matrix) within the first 2 hours to incorporate the early watch signal. After 24 hours, the full retrain incorporates the complete day-one viewing data.

Q: What prevents a popular title from dominating all users’ recommendations regardless of their personal preferences? A: Three mechanisms. First, the ALS training loss penalizes overfitting to popular items via negative sampling - random users who did not watch a popular title provide gradient signal that pushes the popular item’s vector away from the average user vector. Second, the popularity debiasing in Two-Tower training down-weights gradient contributions from popular items when they appear as negatives. Third, the post-ranking diversity filter caps the number of occurrences of any single franchise or genre in the final ranked list. Without these mechanisms, the recommendation engine would converge to showing every user the same top-20 most-watched titles - a popularity leaderboard rather than a personalized homepage.

Interview Questions

Q: How would you design the matrix factorization training pipeline to retrain on 70 billion events every 2 hours without exceeding the available Spark cluster budget? Expected depth: Discuss incremental ALS (warm-starting the next training run from the previous model’s P and Q matrices, running only a few ALS iterations rather than training from scratch), data sampling strategies (full interaction data is not needed - stratified sampling by recency, keeping all events from the last 24 hours and sampling 10% of older events gives 80% of the training benefit at 20% of the compute cost), and the trade-off between training quality (more iterations, more data = better model) and deployment frequency (faster retraining = fresher recommendations). Cover the infrastructure: Spark on EMR with spot instances for the training job, S3 for reading the 7-day event log, and Cassandra writes for publishing the resulting vectors.

Q: Walk me through how you would implement the diversity injection step in the post-ranking filter. Expected depth: Discuss the greedy re-ranking approach - iterate through the top-ranked 75 candidates and select items that maximize both relevance (ranking model score) and diversity (distance from already-selected items in the embedding space). Cover the specific diversity metrics: genre vector cosine distance, franchise deduplication (no two items from the same series in top-5 unless one is “continue watching”), and language diversity for multilingual households. Discuss the tension between diversity and relevance - over-indexing on diversity by pushing the user’s 2nd-favorite genre into the first row hurts play rate. The correct intervention is to ensure genre variety across the full 40-tile homepage, not within each row.

Q: Design the A/B testing framework to support running 50 simultaneous recommendation experiments without them interfering with each other. Expected depth: Discuss the layer isolation approach (candidate retrieval layer, ranking layer, post-filter layer, UI layer - experiments in different layers are statistically independent). Cover user bucketing with deterministic hashing (same user always in same bucket for a given experiment), traffic splitting constraints (the sum of all experiment traffic shares within a layer cannot exceed 100%), holdout groups (5-10% of users are always in the control group for all experiments, providing a baseline), and the Bonferroni correction needed when testing 50 hypotheses simultaneously (p-value threshold should be 0.05/50 = 0.001 to maintain 5% false positive rate across all experiments).

Q: How would you handle the user embedding store going from 140 GB (today) to 1.4 TB (after growing to 2.8 billion users)? Expected depth: Discuss consistent hashing for sharding user embeddings across Cassandra nodes (user_id % N_shards determines which node), read-path caching in Redis (only the ~50M currently active users need sub-millisecond embedding lookups; the rest can be served from Cassandra at 5-10ms), and the option to reduce embedding dimension (64-dim instead of 128-dim) at larger scale since more training data compensates for lower dimensionality. Cover the cold path: users who haven’t logged in in 30+ days don’t need their embeddings in Redis; Cassandra cold reads are acceptable. Discuss the write path for publishing new embeddings after each training run: at 1.4 TB, a full write takes 4-6 hours at 100 Gbps; a partial update strategy (only update users whose vectors changed by more than a threshold) reduces write volume by 80%.