Community研究与数据分析github.com

构建特征流水线

构建机器学习特征流水线和特征存储——通过时间点正确连接避免标签泄露,实现离线/在线一致性、特征新鲜度和回填,并使用 Feast 等工具进行物化。适用于机器学习特征工程、防止训练/推理偏差或数据泄露、搭建特征存储,或为训练回填历史特征。

构建特征流水线 是什么?

该技能专注于构建生产级机器学习特征流水线和特征存储,核心目标是防止标签泄露和训练/推理偏差。它详细说明了时间点正确连接(as-of join)的实现方式,确保在训练时只使用标签时间点之前已知的数据,从而避免标签泄露这一最具破坏性的特征错误。同时强调离线(训练)和在线(推理)特征计算的单一共享定义,通过 Feast 等工具实现离线存储和在线存储双路物化,消除因逻辑分歧导致的训练/推理偏差。技能还涵盖了在线特征新鲜度监控、物化调度以及历史特征幂等回填的最佳实践。支持 SQL 和 Feast 等技术平台,适用于使用 dbt、Spark 进行数据处理后需要为 ML 模型准备特征的场景,以及需要搭建特征存储或进行历史回填的团队。

兼容平台~Claude Code~Codex CLI~Cursor
npx skills add https://github.com/Unknown-333/awesome-data-engineering-skills/tree/main/skills/building-feature-pipelines

Installed? Explore more 研究与数据分析 skills: affaan-m/uspto-database, affaan-m/scholar-evaluation, affaan-m/literature-review · View all 6 →

在你喜欢的 AI 中提问

打开一个已预加载此 Agent Skill 的新对话。

文档

构建特征流水线 是做什么的?

使用时机

  • 从仓库/流数据中为机器学习模型工程化特征。
  • 防止标签泄露和训练/推理偏差。
  • 搭建特征存储、在线推理或历史回填。
  • 除非为机器学习特征提供输入,否则不要用于通用建模/聚合(请使用 dbt/Spark 技能)。

工作流程

- [ ] 为每个特征定义实体键和事件时间戳
- [ ] 使用时间点正确连接(以标签时间为准)构建训练集
- [ ] 为离线(训练)和在线(推理)**共享一个定义**
- [ ] 为在线特征设置新鲜度/物化策略
- [ ] 幂等地回填历史特征以用于训练
  1. 时间点正确性。 以每个标签的时间戳为准连接特征——只使用预测时间之前已知的数据。这可以防止标签泄露,这是最具破坏性的特征错误。
  2. 离线/在线一致性。 以相同方式计算训练(离线、批处理)和推理(在线、低延迟)的特征。逻辑分歧会导致训练/推理偏差以及无提示的生产性能下降。
  3. 新鲜度。 在线特征必须按照模型可接受的数据陈旧容忍度进行调度物化。
  4. 幂等回填。 重新计算历史特征必须是可重复的(参见 designing-backfills-and-replays)。

模式

时间点(as-of)连接——选在每条标签事件之前最新的特征值:

SELECT l.entity_id, l.label_ts, f.value AS feature
FROM labels l
LEFT JOIN features f
  ON f.entity_id = l.entity_id
 AND f.event_ts <= l.label_ts          -- 仅使用过去的数据;无泄露
QUALIFY ROW_NUMBER() OVER (
  PARTITION BY l.entity_id, l.label_ts ORDER BY f.event_ts DESC) = 1;

单一定义,双路路径——将特征定义一次(例如 Feast FeatureView);将其物化到离线存储用于训练,物化到在线存储用于推理,这样两者使用相同的逻辑。

新鲜度 + 物化——调度在线物化;像监控任何数据集 SLA 一样监控特征新鲜度(implementing-pipeline-observability)。

常见陷阱

  • 标签泄露——连接标签时间之后计算的特征会虚增离线指标并在生产中崩溃;始终使用 as-of 连接。
  • 训练/推理偏差——离线实现和在线实现分开会导致漂移;共享一个定义。
  • 过时的在线特征——模型基于旧值进行推理;设置并监控新鲜度。
  • 不可重现的回填——训练历史不一致;确保特征重新计算是幂等的。
  • 缺少实体/时间戳键——特征无法跨时间正确连接;两者都需要。
  • 未版本化的特征——无提示地重新定义会破坏模型可比性;对特征定义进行版本控制。

Individual skills in this repo

This repo contains 9 individual skills — each has its own dedicated page.

Unknown-333/authoring-airflow-dags

Write production-grade Apache Airflow DAGs using the TaskFlow API — idempotent tasks, correct scheduling and catchup, retries/SLAs, connections/variables, and avoiding top-level code. Use when creating or reviewing Airflow DAGs, scheduling pipelines, wiring task dependencies, configuring retries/backfills, or fixing non-idempotent tasks.

Unknown-333/building-dagster-assets

Build Dagster pipelines using software-defined assets — asset dependencies, partitions, resources and IO managers, asset checks, and schedules/sensors. Use when creating Dagster assets or jobs, modeling data as assets, adding partitions or backfills, wiring resources/IO managers, or migrating from task-based orchestration to assets.

Unknown-333/building-dbt-models

Build well-structured dbt models — staging/intermediate/marts layers, ref() and source(), materializations, and incremental models with the right strategy. Use when creating or refactoring dbt models, choosing table vs view vs incremental, structuring a dbt project, or writing incremental logic.

Unknown-333/building-iceberg-tables

Design and operate Apache Iceberg tables — partitioning and hidden partitioning, partition/schema evolution, snapshots and time travel, compaction and small-file cleanup, and MERGE/upsert for lakehouse tables on Spark, Flink, Trino, or Snowflake. Use when creating or maintaining Iceberg tables, choosing partitioning, evolving schema/partitions, or fixing small-file and metadata bloat.

Unknown-333/building-ingestion-pipelines

Build batch and incremental data ingestion (extract-load) pipelines — full vs incremental extraction, change data capture (CDC), watermarks and high-water marks, API pagination and rate limits, and choosing managed EL tools (Fivetran, Airbyte) vs custom code. Use when ingesting data from databases, APIs, files, or SaaS into a warehouse/lake, or designing incremental extraction and CDC.

Unknown-333/building-kafka-consumers

Build reliable Apache Kafka consumers and producers — consumer groups and partition assignment, offset commit strategy, at-least-once vs exactly-once, idempotent/transactional producers, rebalancing, and dead-letter handling. Use when writing Kafka consumers/producers, configuring offset commits or consumer groups, tuning throughput, or handling rebalances and poison messages.

Unknown-333/debugging-data-pipelines

Systematically root-cause data pipeline failures and data incidents — job errors, wrong or missing data, duplicates, and freshness misses — by tracing lineage upstream, isolating the failing stage, reconciling against source, and planning a safe fix and backfill. Use when a pipeline fails, numbers look wrong, data is missing or duplicated, a dashboard is stale, or a stakeholder reports a data discrepancy.

Unknown-333/designing-backfills-and-replays

Plan and run safe data backfills and replays — idempotent reprocessing of historical windows, partition-by-partition execution, isolating backfill compute from production, verifying results, and avoiding double-counting or changed history. Use when backfilling a new or fixed model, reprocessing after a bug, replaying events, or loading history for a new pipeline without corrupting existing data.

Unknown-333/designing-data-contracts

Define and enforce data contracts between producers and consumers — explicit schema, semantics, ownership, SLAs, and versioning — to prevent silent upstream changes from breaking downstream pipelines. Use when a producer schema change could break consumers, defining an interface between teams/services and the warehouse, or adding schema enforcement at ingestion.

相关技能