AI Feature Store Architecture, in depth: registry, offline and online stores, point-in-time joins and skew

What a feature store is, and what it is not

Strip away the product names and a feature store does four jobs. It defines features: a name, an entity key such as card_id, a type, a source, a freshness budget and an owner. It stores history with event timestamps so you can ask what a value was at any past moment. It serves the present from a key-value store indexed by entity. And it describes itself: lineage, access rules and monitoring hang off the registry.

What it does not do is guarantee correctness on its own. Many teams compute features in a Spark job outside the store, write the results in, and compute the same feature again inside the request handler from live data. The store then faithfully serves two different numbers under one name. Skew disappears only when there is a single transformation whose output lands in both stores, or when the store itself executes the logic in both contexts, which is what on-demand transformations are for. Ask of every feature: where does the code that computes it live, and how many copies of that code exist? One copy is the goal.

Reuse is the second payoff. A well-named card_txn_count_1h written once, documented in the registry and discoverable by search, saves every later team from rewriting it slightly differently.

What a feature store is, and what it is not

Strip away the product names and a feature store does four jobs. It defines features: a name, an entity key such as card_id, a type, a source, a freshness budget and an owner. It stores history with event timestamps so you can ask what a value was at any past moment. It serves the present from a key-value store indexed by entity. And it describes itself: lineage, access rules and monitoring hang off the registry.

What it does not do is guarantee correctness on its own. Many teams compute features in a Spark job outside the store, write the results in, and compute the same feature again inside the request handler from live data. The store then faithfully serves two different numbers under one name. Skew disappears only when there is a single transformation whose output lands in both stores, or when the store itself executes the logic in both contexts, which is what on-demand transformations are for. Ask of every feature: where does the code that computes it live, and how many copies of that code exist? One copy is the goal.

Reuse is the second payoff. A well-named card_txn_count_1h written once, documented in the registry and discoverable by search, saves every later team from rewriting it slightly differently.

The components

The registry is the contract. In Feast it is a file or database that feast apply populates from Python definitions kept in Git. Pipelines, the offline query engine and the online server all read it, so a schema change is a reviewed code change rather than a surprise.

The offline store is a warehouse or lake table (BigQuery, Snowflake, Redshift, Spark over Parquet or Delta). It is append-only and keeps every value with its event_timestamp and often a created timestamp that records when the row was written, which breaks ties when a late correction arrives for the same event time.

The online store keeps one row per entity per feature view: the most recent value. Redis, DynamoDB, Bigtable, Cassandra and Postgres are common choices. It is keyed for point reads, sized to fit the active entity set, and usually expires rows after a TTL so stale values read as missing rather than wrong.

Data reaches the online store in two ways. Materialization copies the latest values from the offline store on a schedule. Push writes fresh values directly from a stream job. A serving layer, either an SDK call in your service or a feature server, resolves a request for named features into key lookups and returns a vector.

Point-in-time joins: training data without leakage

Point-in-time correctness is the property that separates a feature store from a table of aggregates. Each training example has an entity key, an event time and a label. For each one you need the feature value that was known at that time, not the value today. Join on key alone and a churn model learns from purchase counts computed after the customer churned: offline metrics look excellent and production collapses.

The rule Feast applies: for each entity row with timestamp T, take the latest feature row for the same key with event_timestamp <= T and event_timestamp > T - ttl. Older rows are treated as missing. The same logic in pandas, which is useful for testing and small data:

import pandas as pd

def point_in_time_join(labels, feats, key, ttl):
    """labels: key, event_ts, label.  feats: key, event_ts, created_ts, value columns."""
    feats = (feats.sort_values(["event_ts", "created_ts"])
                  .drop_duplicates([key, "event_ts"], keep="last"))   # late corrections win
    out = pd.merge_asof(
        labels.sort_values("event_ts"),
        feats.sort_values("event_ts").rename(columns={"event_ts": "feature_ts"}),
        left_on="event_ts", right_on="feature_ts",
        by=key, direction="backward",        # never look forward in time
        tolerance=ttl, allow_exact_matches=True)
    return out.drop(columns=["created_ts"])

In a warehouse the equivalent is a join on key with feature_ts <= label_ts followed by ROW_NUMBER() OVER (PARTITION BY label_id ORDER BY feature_ts DESC) and a filter on row one. That query is expensive on large label sets, so production engines bucket by time or sort-merge; the semantics do not change. One subtlety: the timestamp must reflect when the value became available. A daily aggregate stamped at midnight but landed at 06:00 leaks six hours unless you stamp it with the landing time.

Defining and reading features in Feast

Here is a minimal Feast repository for card features. Definitions live in Git; feast apply registers them, feast materialize-incremental fills the online store, and the same names are used for training and serving.

from datetime import timedelta
from feast import Entity, FeatureView, Field, FileSource
from feast.types import Float32, Int64

card = Entity(name="card", join_keys=["card_id"])

card_daily_source = FileSource(
    name="card_daily_stats",
    path="s3://feature-lake/card_daily_stats/",
    timestamp_field="event_timestamp",
    created_timestamp_column="created",
)

card_daily = FeatureView(
    name="card_daily",
    entities=[card],
    ttl=timedelta(days=2),          # older than this reads as missing
    schema=[
        Field(name="avg_amount_30d", dtype=Float32),
        Field(name="distinct_merchants_30d", dtype=Int64),
    ],
    online=True,
    source=card_daily_source,
    tags={"owner": "risk-features"},
)

from feast import FeatureStore
store = FeatureStore(repo_path=".")

# Training: entity_df has card_id, event_timestamp, label
train_df = store.get_historical_features(
    entity_df=labels_df,
    features=["card_daily:avg_amount_30d", "card_daily:distinct_merchants_30d"],
).to_df()

# Serving: one request, current values
vec = store.get_online_features(
    features=["card_daily:avg_amount_30d", "card_daily:distinct_merchants_30d"],
    entity_rows=[{"card_id": 4417}],
).to_dict()

Batch, streaming and on-demand features

Features fall into three freshness tiers, and each tier has its own pipeline shape. Batch features (30-day averages, account age) are computed hourly or daily in SQL or Spark, appended offline and materialized online. Their freshness is the schedule plus job runtime. Streaming features (transactions in the last hour) are maintained by a stream job with windowed state and pushed to the online store within seconds; the same job must also append to the offline store, or training will never see the values serving used. On-demand features depend on the request itself, such as the ratio of this transaction's amount to the 30-day average. They are computed at request time from request data plus stored features, and the store replays the same function over historical rows when building training sets.

Streaming is where skew sneaks back in. A Flink job computing a sliding one-hour count and a backfill SQL query computing the same count over history are two implementations. They differ on late events, window boundaries and deduplication. Either log the values the stream actually produced and train on those logs, or run the same engine in batch mode for backfills, and compare the two outputs on an overlapping day before trusting either.

Worked example: fraud scoring in 10 milliseconds

A card issuer scores each authorization in under 50 ms end to end and gives the feature lookup 10 ms of that. The model uses avg_amount_30d and distinct_merchants_30d (batch, daily), txn_count_1h (streaming), and amount_vs_avg (on-demand).

1. Training: risk analysts produce labels (card, authorization time, fraud flag) for six months. get_historical_features joins each to the feature values as of that authorization. Rows with missing streaming values are kept and flagged, because serving will also see missing values on a cold key.
2. Online data: 40 million active cards, three stored features, roughly 150 bytes per entity including key and overhead in Redis, so about 6 GB before replication. A daily materialization writes the batch view; the stream job pushes txn_count_1h on every transaction.
3. Serving: the scoring service sends one batched lookup for the card, receives three values, computes amount_vs_avg from the request amount, and calls the model. Redis p99 on a single key is about a millisecond inside the same zone, leaving room for the network.
4. Logging: the service writes the exact feature vector it used, with a request id, to a log table. Joining those logs to later labels gives training data that is skew-free by construction and a direct way to measure skew against the offline recomputation.

Failure modes

The failure modes that reach production, roughly in order of how often they hurt:

Distribution drift belongs on the same dashboard: track mean, null rate and a population stability index per feature, compared between the training window and the last day of served values. See drift detection for the statistics.

Operating the store

Run the store like the production database it is. For each feature view publish four numbers: freshness (now minus newest event timestamp online), online read latency p50 and p99, null rate, and the skew rate from the served-versus-recomputed comparison. Page on freshness and latency; ticket on null rate and drift.

Treat definitions as code. Changes go through review, feast plan shows what will change before feast apply, and a breaking change (new type, new semantics) gets a new feature name, not an edit in place, so models trained on the old meaning keep working. Delete a feature only after the registry shows no model or feature service still reads it.

Backfills are the expensive operation. Recomputing two years of a streaming feature can cost more than a month of normal running, so size them, run them per partition, and verify a sample against the live values before switching consumers over. The MLOps lifecycle and training pipeline articles cover where these steps sit in a release process, and model serving covers the request path the online store sits behind.

Trade-offs

Open source versus managed. Feast gives you the registry, offline retrieval and serving on infrastructure you already run; you own the pipelines and operations. Managed stores in cloud ML platforms remove that work and tie you to one vendor's storage and pricing. Online store choice trades latency against cost: Redis is fastest and memory-priced, DynamoDB and Bigtable scale with less operational work at a few milliseconds more. Freshness costs money linearly: moving a feature from daily to streaming adds a stateful job, a push path and a second correctness problem, so do it only when an experiment shows the model improves.

When not to build one: a single model with a handful of features computed in the request path, or a batch scoring job that reads one warehouse table, gains little. The store pays off at the second or third model sharing features, or the first time leakage costs you a launch. Version your training sets either way, as described in data versioning.

What to do next

1. List every feature your production models read and, for each, where its computing code lives. Count the copies.
2. Pick the feature with two implementations and add served-vector logging so you can measure the gap.
3. Rebuild one training set with a point-in-time join and compare offline metrics with the old key-only join.
4. Add a TTL and a freshness alert to every online feature view.
5. Define null defaults in the registry and apply them identically in training and serving.
6. Only then decide whether a streaming tier is worth its cost, using an offline experiment on fresher values.