Think of a jazz musician who has just heard one song and must immediately improvise an infinite set that feels coherent - same mood, compatible keys, similar energy arcs - while surprising the audience with each new tune and never repeating the same piece twice across the whole night. That improvisation problem, scaled to 600 million simultaneous listeners each holding a different seed song, is what Spotify’s Song Radio must solve in under 500 milliseconds per request.

The technical challenge compounds across several dimensions at once. The catalog contains 80 million tracks. Each listener has a unique taste profile built from years of streaming history. A song played in this session must not replay for five sessions back. And the system must produce a result that feels neither monotonous - all songs identical to the seed - nor jarring, where genre whiplash breaks the listener’s flow. This is not a simple similarity search problem; it is a multi-objective optimization problem running at internet scale.

Three tensions define the design space. Relevance versus diversity: a playlist of 50 songs acoustically identical to the seed is technically accurate but boring after five tracks. The system needs controlled variety. Personalization versus latency: computing a bespoke recommendation for each of 600 million users in real time requires either precomputed embeddings or extremely fast model inference. You cannot run a deep neural network per user per request. And cold-start for new releases: a track uploaded this morning has zero listening history, so collaborative filtering signals - which depend entirely on aggregated play counts - cannot apply. Audio features are the only signal available, and the system must degrade gracefully to pure acoustic similarity when history is absent.

Requirements and Constraints

Functional Requirements

• Given a seed track_id and user_id, return an ordered playlist of at least 50 recommended tracks in real time
• The recommendation mix must blend acoustic similarity (audio features) with collaborative filtering signals (user listening patterns)
• Track deduplication must span at least 5 previous radio sessions for the same user - no track repeats within that window
• New tracks with zero play history must receive recommendations via acoustic similarity fallback
• The playlist must inject diversity to prevent genre or tempo monotony across the full session

Non-Functional Requirements

• Latency: sub-500ms end-to-end for the first batch of recommendations; P99 under 400ms
• Throughput: support 600 million active users; assume 10 million concurrent radio sessions at peak
• Catalog scale: 80 million tracks indexed for approximate nearest neighbor search
• Session memory: deduplication across 5 most recent radio sessions per user
• Availability: 99.99% - radio is a high-engagement feature; degraded mode serves pure audio similarity results when the CF model is unavailable

Constraints and Scope

• Track audio analysis (raw audio to feature vector) is an offline batch pipeline; we receive pre-computed feature vectors at query time
• Explicit user feedback (thumbs up/down) is ingested asynchronously and does not affect the current session in real time
• Ad insertion and podcast episodes are handled by separate systems outside this scope
• The Diversity Engine operates on the top-200 post-scored candidates, not the raw 80 million catalog

High-Level Architecture

The system has six layers. The API Gateway accepts the seed track_id and user_id, validates auth, and routes to the Radio Generation Service - the orchestrator that calls every downstream component. The Feature Store provides pre-computed audio embeddings for the seed track; these are fed into the ANN Index for approximate nearest neighbor retrieval of 500 candidate tracks. In parallel, the CF Model scores those candidates against the user’s taste profile. The Dedup Store and Diversity Engine then filter and reorder the scored candidates before the final playlist is returned to the client.

Audio Feature Extraction

Every track in Spotify’s catalog passes through an audio analysis pipeline that converts raw audio into a 128-dimensional feature vector. The low-level features include tempo (beats per minute), key (musical key and mode), valence (musical positiveness, 0 to 1), energy (perceptual intensity), and danceability (how suitable a track is for dancing). These five scalar features are visible via Spotify’s public API, but the internal representation is a much richer learned embedding from a neural audio encoder.