Picture an air traffic control room next to its incident investigation archive. The controller’s radar screen has to show a plane’s current position within a second or two, refreshed constantly, discarded the moment it is stale, because the only thing that matters is right now. Months later, when investigators reconstruct a near-miss, they do not want that live feed. They want an immutable record of exactly what the controller’s screen showed at 14:32:07, not data that trickled in a minute afterward and quietly rewrote history. Both views describe the same airspace, but they are built for opposite jobs, one optimized for speed and recency, the other for completeness and precise timing, and if the two ever disagree about where a plane was, someone has a very bad day.

A real-time ML feature store is that control room and that archive, serving the same numbers for two audiences with irreconcilable requirements. An online payments platform running fraud and recommendation models needs a feature vector, a user’s transaction velocity, a merchant’s chargeback rate, a device’s recent login pattern, delivered to the scoring service in under 5 milliseconds at checkout time, because nobody waits three seconds for a card to authorize. The team training tomorrow’s fraud model needs the opposite: a precisely time-stamped history of every feature value that ever existed, so a training row labeled “fraudulent” from six weeks ago is joined only to the feature values that were actually known at that exact moment, not to information that arrived later and would let the model cheat by seeing the future.

The naive approach is to let each team solve this independently. A data scientist computes “average transaction amount over the last 7 days” in a notebook against the warehouse, ships a model, and an engineer reimplements the same logic in the online serving path using whatever the production database happens to expose. The two implementations round differently, bucket time zones differently, or handle nulls differently, and the model that scored beautifully offline degrades in production for reasons nobody can quite pin down, a phenomenon with a name: training-serving skew. Querying the transactional database directly at inference time fails for a more basic reason, it was built for row lookups and writes, not for assembling a 40-feature vector under a 5-millisecond budget while a payments system is also trying to write to the same tables. And naively joining historical events to training labels by “just take the latest value” silently leaks the future into the past, producing a model that looks accurate in every offline metric and then fails the moment it meets data it could not have seen in advance.

The forces in tension are latency against completeness, and consistency against duplicated logic. We need to solve for a serving path that answers a feature lookup in single-digit milliseconds without ever touching a raw event log, a point-in-time correct join capability that can reconstruct exactly what was known at any past moment without leaking the future, a single shared definition of every feature so the number computed for training and the number computed for serving are the same code path rather than two, and a materialization pipeline that keeps the fast path fresh within a bounded, monitored staleness budget.

Requirements and Constraints

Functional Requirements

• Register every feature in a central feature registry with its name, owning team, entity type, data type, and a versioned reference to the exact transformation logic that produces it
• Serve a pre-computed feature vector for one or more entity IDs from a low-latency online store, given a model’s declared list of required features
• Materialize features from streaming and batch sources into both the online store and the offline store using the same transformation logic, on a schedule or trigger defined per feature group
• Support point-in-time correct joins: given a set of labeled rows with entity IDs and timestamps, return the feature values that were actually known as of each timestamp, with no leakage from future or late-arriving data
• Pin a specific feature version to a specific trained model so that model can be scored consistently months or years later, even after the feature’s definition evolves
• Detect and alert on training-serving skew by comparing the statistical distribution of features as computed offline against the same features as observed in the online store
• Track feature freshness against a per-feature-group service level objective and alert when materialization falls behind

Non-Functional Requirements

• Scale: 50 million tracked entities (users, merchants, devices), 5,000 registered features across 200 feature groups, roughly 10,000 feature-update events per second sustained via streaming ingestion
• Online read latency: p99 under 5ms and p50 under 1ms for a feature vector lookup, at a sustained peak of 300,000 requests per second, each request reading around 20 features
• Freshness: a default staleness SLO of 5 minutes for batch-derived feature groups, configurable down to 30 seconds for streaming feature groups used in fraud scoring
• Offline join throughput: construct a training dataset joining 500 million labeled rows against 40 features per row within a 4-hour nightly batch window
• Availability: 99.99% for the online serving path; the offline path can tolerate a few hours of degraded availability without affecting live inference
• Consistency: feature values computed for training and for serving must be bit-for-bit identical for the same entity, timestamp, and feature version, measured by a continuous skew-detection job
• Retention: the offline feature log is retained for at least 2 years to support retraining and point-in-time audits of past model decisions

Constraints and Assumptions

• We are not building a general-purpose stream processing engine; the pipeline orchestrator runs on top of an existing engine such as Flink or Spark Structured Streaming
• We are not building the model training loop, the model registry, or the experiment tracking system; this design covers only the feature layer those systems consume
• We assume every feature’s transformation logic is expressed once, in a shared definition that both the streaming and batch execution paths can run unmodified, rather than being reimplemented per path
• Feature discovery tooling (a searchable UI/catalog on top of the registry) is a thin consumer of the registry API and is out of scope for this design
• We assume upstream event sources (transaction streams, clickstreams, warehouse tables) already exist and are reliable; the interfaces that produce those raw events are out of scope

High-Level Architecture

The system has six major components: the Feature Registry, the Feature Pipeline Orchestrator, the Online Feature Store, the Offline Feature Store, the Point-in-Time Join Engine, and the client services that consume them, an Online Inference Service and offline Training Jobs.

A raw event, a checkout transaction, a page view, a device fingerprint, arrives from an event source and reaches the Feature Pipeline Orchestrator, which executes the transformation logic defined in the Feature Registry to compute one or more feature values. That single computation is dual-written in the same pipeline run to the Online Feature Store, a low-latency key-value layer keyed by entity ID, and to the Offline Feature Store, an append-only, time-partitioned log that never overwrites a prior value. The Online Inference Service reads a feature vector for an entity directly from the Online Feature Store in single-digit milliseconds at request time. Separately, when a training job needs a labeled dataset, it hands a set of entity IDs and label timestamps to the Point-in-Time Join Engine, which reconstructs, for every label, exactly the feature values that existed at that moment by querying the Offline Feature Store.

The Feature Registry sits beside all of this as the source of truth, not in the data path itself. Every feature the pipeline materializes, every read the inference service issues, and every join the training path performs resolves its feature definition and version through the registry first. That single reference point is what keeps a feature named txn_amount_sum_1h meaning the same computed value everywhere it appears.

Component Deep Dives

The Feature Registry

This component’s job is to be the single, versioned source of truth for what a feature means, who owns it, and which exact transformation logic produced it.

A reasonable-sounding shortcut is to assume a feature name is enough, if two pipelines both compute something called avg_txn_amount_7d, they must be computing the same thing. In practice they usually are not. One team’s notebook buckets by UTC day, another team’s streaming job buckets by a sliding 7-day window with a different rounding rule, and both call the result the same name. The registry fixes this by making every feature reference a specific feature_version, not just a name, and by storing a content hash of the transformation logic itself, so a silent code change is detectable even if nobody updates the version number.

-- Feature registry: definitions, ownership, and versioned transformation logic
CREATE TABLE feature_groups (
    feature_group_id      BIGSERIAL PRIMARY KEY,
    name                   TEXT NOT NULL UNIQUE,
    entity_type            TEXT NOT NULL,
    owner_team             TEXT NOT NULL,
    source_type            TEXT NOT NULL CHECK (source_type IN ('stream', 'batch', 'on_demand')),
    freshness_slo_seconds   INTEGER NOT NULL DEFAULT 300,
    created_at              TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE TABLE features (
    feature_id           BIGSERIAL PRIMARY KEY,
    feature_group_id     BIGINT NOT NULL REFERENCES feature_groups(feature_group_id),
    name                  TEXT NOT NULL,
    dtype                 TEXT NOT NULL CHECK (dtype IN ('int64', 'float64', 'bool', 'string', 'embedding')),
    description           TEXT NOT NULL DEFAULT '',
    UNIQUE (feature_group_id, name)
);

CREATE TABLE feature_versions (
    version_id            BIGSERIAL PRIMARY KEY,
    feature_id            BIGINT NOT NULL REFERENCES features(feature_id),
    version_number         INTEGER NOT NULL,
    transformation_ref      TEXT NOT NULL,
    transformation_hash     CHAR(64) NOT NULL,
    is_active               BOOLEAN NOT NULL DEFAULT TRUE,
    published_at            TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    UNIQUE (feature_id, version_number)
);
CREATE INDEX ON feature_versions (feature_id) WHERE is_active;

-- Pins a model to the exact feature version it was trained against
CREATE TABLE model_feature_bindings (
    binding_id            BIGSERIAL PRIMARY KEY,
    model_name             TEXT NOT NULL,
    model_version           TEXT NOT NULL,
    feature_id              BIGINT NOT NULL REFERENCES features(feature_id),
    version_id               BIGINT NOT NULL REFERENCES feature_versions(version_id),
    bound_at                 TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    UNIQUE (model_name, model_version, feature_id)
);

Without versioning, retraining a model six months later against the “current” definition of a feature silently changes what that feature means, and nobody can explain why a model that used to work now behaves differently. model_feature_bindings is what lets an old model keep scoring correctly even as the feature catalog evolves underneath it, because the binding pins a specific version_id, not a name that can quietly drift.