Unknown-333/building-feature-pipelines

Building Feature Pipelines

When to use

• Engineering features for ML models from warehouse/stream data.
• Preventing label leakage and train/serve skew.
• Setting up a feature store, online serving, or historical backfills.
• Do NOT use for general modeling/aggregation (use dbt/Spark skills) unless it feeds ML features.

Workflow

- [ ] Define each feature with an entity key and an event timestamp
- [ ] Build training sets with point-in-time-correct joins (as-of the label time)
- [ ] Share ONE definition for offline (training) and online (serving)
- [ ] Set freshness/materialization for online features
- [ ] Backfill historical features idempotently for training

1. Point-in-time correctness. Join features as of each label's timestamp — use only data that was known before the prediction time. This prevents label leakage, the most damaging feature bug.
2. Offline/online parity. Compute a feature the same way for training (offline, batch) and serving (online, low-latency). Divergent logic causes train/serve skew and silent production degradation.
3. Freshness. Online features must be materialized on a schedule that meets the model's staleness tolerance.
4. Idempotent backfills. Recomputing historical features must be repeatable (see designing-backfills-and-replays).

Patterns

Point-in-time (as-of) join — pick the latest feature value strictly before each label event:

SELECT l.entity_id, l.label_ts, f.value AS feature
FROM labels l
LEFT JOIN features f
  ON f.entity_id = l.entity_id
 AND f.event_ts <= l.label_ts          -- only past data; no leakage
QUALIFY ROW_NUMBER() OVER (
  PARTITION BY l.entity_id, l.label_ts ORDER BY f.event_ts DESC) = 1;

Single definition, two paths — define the feature once (e.g. Feast FeatureView); materialize to an offline store for training and an online store for serving so both use identical logic.

Freshness + materialization — schedule online materialization; monitor feature freshness like any dataset SLA (implementing-pipeline-observability).

Common pitfalls

• Label leakage — joining features computed after the label time inflates offline metrics and collapses in production; always as-of join.
• Train/serve skew — separate offline and online implementations drift; share one definition.
• Stale online features — model serves on old values; set and monitor freshness.
• Non-reproducible backfills — inconsistent training history; make feature recomputation idempotent.
• No entity/timestamp keys — features can't be joined correctly across time; require both.
• Unversioned features — silent redefinition breaks model comparability; version feature definitions.