CommunityPesquisa e Análise de Dadosgithub.com

Unknown-333/designing-data-contracts

Define and enforce data contracts between producers and consumers — explicit schema, semantics, ownership, SLAs, and versioning — to prevent silent upstream changes from breaking downstream pipelines. Use when a producer schema change could break consumers, defining an interface between teams/services and the warehouse, or adding schema enforcement at ingestion.

O que é designing-data-contracts?

designing-data-contracts is a Claude Code agent skill that define and enforce data contracts between producers and consumers — explicit schema, semantics, ownership, SLAs, and versioning — to prevent silent upstream changes from breaking downstream pipelines. Use when a producer schema change could break consumers, defining an interface between teams/services and the warehouse, or adding schema enforcement at ingestion.

Funciona com~Claude Code~Codex CLI~Cursor
npx skills add https://github.com/Unknown-333/awesome-data-engineering-skills/tree/main/skills/designing-data-contracts

Installed? Explore more Pesquisa e Análise de Dados skills: obra/superpowers, affaan-m/quarkus-verification, affaan-m/uspto-database · View all 6 →

Ver original→Navegar por todas as habilidades

Perguntar na sua IA favorita

Abre um novo chat com esta habilidade de agente já pré-carregada.

OpenAIChatGPTClaudeClaudeGeminiGeminiGrokGrokPerplexityPerplexityDeepSeekDeepSeek

Documentação

Designing Data Contracts

When to use

  • An upstream (service, event, API, file) feeds downstream pipelines and a change could break them silently.
  • Defining the interface between a producing team/system and the warehouse.
  • Adding schema/quality enforcement at the ingestion boundary.
  • Do NOT use for internal model-to-model changes within one dbt project (use tests + handling-schema-evolution).

What a contract specifies

  • Schema: fields, types, nullability, and allowed values.
  • Semantics: what each field means and its unit/grain.
  • Guarantees: freshness/SLA, volume expectations, uniqueness of keys.
  • Ownership: who produces it and who to contact.
  • Versioning + change policy: how breaking changes are communicated.

Workflow

- [ ] Write the contract as a versioned, checked-in schema (not tribal knowledge)
- [ ] Enforce it at the ingestion boundary (validate on arrival)
- [ ] Classify changes: additive (safe) vs breaking (needs a new version)
- [ ] On violation, reject/quarantine and alert the producer
- [ ] Version and communicate breaking changes ahead of time
  1. Make it explicit and versioned. Store the contract as code (JSON Schema, Avro/Protobuf schema, or a YAML spec) next to the pipeline, reviewed like any API.
  2. Enforce at the boundary. Validate incoming data against the contract on arrival; reject or quarantine violations instead of loading them.
  3. Classify changes. Additive/optional fields = backward compatible. Removing fields, renaming, tightening types/nullability = breaking → new version.
  4. Fail loudly to the producer, not silently downstream.

Patterns

Contract as JSON Schema (enforced on ingest):

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "required": ["order_id", "amount", "ordered_at"],
  "properties": {
    "order_id": { "type": "string" },
    "amount": { "type": "number", "minimum": 0 },
    "ordered_at": { "type": "string", "format": "date-time" },
    "coupon": { "type": ["string", "null"] }
  },
  "additionalProperties": false
}

Enforcement point — validate each record on ingest; route failures to a quarantine location with the reason, and alert the producing team. This turns a silent downstream break into an immediate, owned signal at the source.

Schema registry (Kafka/Avro) — enforce compatibility (BACKWARD) at publish time so producers cannot ship an incompatible schema.

Common pitfalls

  • Contract as documentation only — if it isn't enforced in code, it drifts and breaks silently.
  • Enforcing deep in the warehouse — catch violations at the boundary, before bad data spreads.
  • No versioning — every change becomes an emergency; version and deprecate gracefully.
  • additionalProperties unrestricted when you need strictness — unexpected fields slip through; set it false where appropriate.
  • No owner — a rejected batch with no one to call stalls the pipeline.
  • Breaking changes with no lead time — coordinate producer/consumer via versioned schemas and a deprecation window.

Individual skills in this repo

This repo contains 9 individual skills — each has its own dedicated page.

Unknown-333/authoring-airflow-dags

Write production-grade Apache Airflow DAGs using the TaskFlow API — idempotent tasks, correct scheduling and catchup, retries/SLAs, connections/variables, and avoiding top-level code. Use when creating or reviewing Airflow DAGs, scheduling pipelines, wiring task dependencies, configuring retries/backfills, or fixing non-idempotent tasks.

Unknown-333/building-dagster-assets

Build Dagster pipelines using software-defined assets — asset dependencies, partitions, resources and IO managers, asset checks, and schedules/sensors. Use when creating Dagster assets or jobs, modeling data as assets, adding partitions or backfills, wiring resources/IO managers, or migrating from task-based orchestration to assets.

Unknown-333/building-dbt-models

Build well-structured dbt models — staging/intermediate/marts layers, ref() and source(), materializations, and incremental models with the right strategy. Use when creating or refactoring dbt models, choosing table vs view vs incremental, structuring a dbt project, or writing incremental logic.

Unknown-333/building-feature-pipelines

Build ML feature pipelines and feature stores — point-in-time-correct joins to avoid label leakage, offline/online parity, feature freshness and backfills, and materialization with tools like Feast. Use when engineering features for ML, preventing train/serve skew or data leakage, building a feature store, or backfilling historical features for training.

Unknown-333/building-iceberg-tables

Design and operate Apache Iceberg tables — partitioning and hidden partitioning, partition/schema evolution, snapshots and time travel, compaction and small-file cleanup, and MERGE/upsert for lakehouse tables on Spark, Flink, Trino, or Snowflake. Use when creating or maintaining Iceberg tables, choosing partitioning, evolving schema/partitions, or fixing small-file and metadata bloat.

Unknown-333/building-ingestion-pipelines

Build batch and incremental data ingestion (extract-load) pipelines — full vs incremental extraction, change data capture (CDC), watermarks and high-water marks, API pagination and rate limits, and choosing managed EL tools (Fivetran, Airbyte) vs custom code. Use when ingesting data from databases, APIs, files, or SaaS into a warehouse/lake, or designing incremental extraction and CDC.

Unknown-333/building-kafka-consumers

Build reliable Apache Kafka consumers and producers — consumer groups and partition assignment, offset commit strategy, at-least-once vs exactly-once, idempotent/transactional producers, rebalancing, and dead-letter handling. Use when writing Kafka consumers/producers, configuring offset commits or consumer groups, tuning throughput, or handling rebalances and poison messages.

Unknown-333/debugging-data-pipelines

Systematically root-cause data pipeline failures and data incidents — job errors, wrong or missing data, duplicates, and freshness misses — by tracing lineage upstream, isolating the failing stage, reconciling against source, and planning a safe fix and backfill. Use when a pipeline fails, numbers look wrong, data is missing or duplicated, a dashboard is stale, or a stakeholder reports a data discrepancy.

Unknown-333/designing-backfills-and-replays

Plan and run safe data backfills and replays — idempotent reprocessing of historical windows, partition-by-partition execution, isolating backfill compute from production, verifying results, and avoiding double-counting or changed history. Use when backfilling a new or fixed model, reprocessing after a bug, replaying events, or loading history for a new pipeline without corrupting existing data.

Habilidades Relacionadas

obra/superpowers

Agent skill repository discovered by 10x-chat research.

community

affaan-m/quarkus-verification

Verification loop for Quarkus projects: build, static analysis, tests with coverage, security scans, native compilation, and diff review before release or PR.

community

affaan-m/uspto-database

USPTO patent and trademark data workflow for official record lookup, PatentSearch queries, TSDR checks, assignment data, and reproducible IP research logs.

community

affaan-m/scholar-evaluation

Structured scholarly-work evaluation for papers, proposals, literature reviews, methods sections, evidence quality, citation support, and research-writing feedback.

community

affaan-m/literature-review

Systematic literature-review workflow for academic, biomedical, technical, and scientific topics, including search planning, source screening, synthesis, citation checks, and evidence logging.

community

affaan-m/research-ops

Evidence-first current-state research workflow for ECC. Use when the user wants fresh facts, comparisons, enrichment, or a recommendation built from current public evidence and any supplied local context.

community
← More Pesquisa e Análise de Dados skills