Communitygithub.com

Airflow DAG Orchestrator

Apache Airflow DAG、オペレーター、SLA監視、ワークフローオーケストレーションを支援します。Airflow、DAG、オペレーター、センサー、スケジューラー、タスク依存関係、SLA、バックフィル、XComに関連するタスクに最適です。dbt変換(使用するのはdbt-analytics-engineer)やストリーミングパイプライン(使用するのはstreaming-pipeline-architect)には適していません。

Airflow DAG Orchestrator とは?

このスキルは、Apache Airflowを使用した信頼性の高いデータパイプラインオーケストレーションを設計・運用するための包括的なガイドです。TaskFlow API(@dag、@taskデコレーター)を使用したPythonネイティブなDAG定義、cronまたはタイムテーブルによるスケジュール設定、タスクのリトライ設定、SLAの追加、ローカルテスト手順など、迅速なスタートアップに必要な基本をカバーします。また、動的タスクマッピング(ファンアウト/ファンイン)やdbtとの統合(cosmosライブラリ推奨)といった高度なアーキテクチャパターンを、実際のコード例を交えて提供します。さらに、過度に大きなタスク、大規模データに対するXComの誤用、誤ったcatchup=True設定、リトライ欠如、DAGファイルのトップレベルコードといったアンチパターンを警告し、品質チェックリストでベストプラクティスを確認できます。このスキルは、Apache Airflow 2.10+、MWAA、Cloud Composer 3に対応し、CeleryExecutorやKubernetesExecutorなどのエグゼキューター、SnowflakeやGoogle Cloudなどのプロバイダー、Prometheusによる監視もサポートしています。開発者やAIエージェントがAirflowパイプラインを効率的に構築・保守するための実践的リファレンスとなります。

対応~Claude Code~Codex CLI~Cursor
npx skills add https://github.com/curiositech/windags-skills/tree/HEAD/skills/airflow-dag-orchestrator

お気に入りのAIに質問する

このエージェントスキルを事前に読み込んだ状態で新しいチャットを開きます。

ドキュメント

Airflow DAG Orchestrator は何をしますか?

適切な依存関係管理、SLA、監視を備えた信頼性の高いデータパイプラインオーケストレーションのためのApache Airflow DAGを設計・運用します。

アクティベーショントリガー

アクティブにする条件: "Airflow", "DAG", "operator", "sensor", "scheduler", "task dependency", "SLA", "backfill", "XCom", "TaskFlow API", "MWAA", "Cloud Composer"

対象外: dbtモデル実行 → dbt-analytics-engineer (ただし、Airflowでdbtをトリガーすることは可能) | ストリーム処理 → streaming-pipeline-architect | ワークフローエンジン (Temporal) → distributed-transaction-manager

クイックスタート

  1. DAGを定義 — TaskFlow API(@dag、@taskデコレーター)を使用してPythonネイティブなDAGを作成
  2. スケジュールを設定 — cronまたはタイムテーブルを使用。バックフィルを意図しない限りcatchup=Falseを設定
  3. リトライを設定 — すべてのタスクにretries=2、retry_delay=timedelta(minutes=5)を設定
  4. SLAを追加 — クリティカルパスのタスクにsla=timedelta(hours=2)を設定
  5. ローカルでテスト — デプロイ前にairflow dags test my_dag 2026-01-01を実行

コア機能

ドメイン技術
AirflowApache Airflow 2.10+, MWAA, Cloud Composer 3
オペレーターBashOperator, PythonOperator, KubernetesPodOperator
プロバイダーapache-airflow-providers-{snowflake, google, aws, dbt-cloud}
エグゼキューターCeleryExecutor, KubernetesExecutor, LocalExecutor
監視SLAミス、タスク実行時間、Airflowメトリクス → Prometheus

アーキテクチャパターン

TaskFlow API DAG

from airflow.decorators import dag, task
from datetime import datetime, timedelta

@dag(
    schedule="0 6 * * *",          # 毎日6:00 UTC
    start_date=datetime(2026, 1, 1),
    catchup=False,
    default_args={
        "retries": 2,
        "retry_delay": timedelta(minutes=5),
        "sla": timedelta(hours=2),
    },
    tags=["finance", "daily"],
)
def daily_revenue_pipeline():

    @task()
    def extract_payments() -> dict:
        """Stripe APIから抽出"""
        data = stripe_client.list_payments(date=today())
        return {"count": len(data), "path": "s3://raw/payments/"}

    @task()
    def extract_orders() -> dict:
        """Shopify APIから抽出"""
        data = shopify_client.list_orders(date=today())
        return {"count": len(data), "path": "s3://raw/orders/"}

    @task()
    def transform(payments: dict, orders: dict) -> str:
        """DuckDBで結合・変換"""
        result_path = run_duckdb_transform(payments["path"], orders["path"])
        return result_path

    @task()
    def load(path: str):
        """Snowflakeにロード"""
        snowflake_copy_into("fct_revenue", path)

    # 関数呼び出しで依存関係を定義
    payments = extract_payments()
    orders = extract_orders()
    transformed = transform(payments, orders)
    load(transformed)

daily_revenue_pipeline()

動的タスクマッピング (ファンアウト/ファンイン)

@task()
def get_partitions() -> list[str]:
    return ["2026-01-01", "2026-01-02", "2026-01-03"]

@task()
def process_partition(partition_date: str) -> dict:
    """パーティションごとに並列実行"""
    return {"date": partition_date, "rows": process(partition_date)}

@task()
def aggregate(results: list[dict]):
    """ファンイン: すべてのパーティション結果を受け取る"""
    total = sum(r["rows"] for r in results)
    log.info(f"合計 {total} 行を処理しました")

# process_partition をすべてのパーティションに動的にマッピング
partitions = get_partitions()
results = process_partition.expand(partition_date=partitions)
aggregate(results)

dbt + Airflow 統合

from airflow.operators.bash import BashOperator
from cosmos import DbtDag, ProjectConfig, ProfileConfig

# オプション1: cosmos (推奨)
dbt_dag = DbtDag(
    project_config=ProjectConfig("/opt/airflow/dbt/"),
    profile_config=ProfileConfig(
        profile_name="default",
        target_name="prod",
    ),
    schedule="@daily",
    dag_id="dbt_daily",
)

# オプション2: BashOperator (シンプル)
dbt_run = BashOperator(
    task_id="dbt_run",
    bash_command="cd /opt/airflow/dbt && dbt build --select tag:daily",
)

アンチパターン

  1. Fat tasks (過度に大きなタスク) — タスクはアトミックな単位であるべき。1つのタスクに抽出+変換+ロードを含めてはいけません
  2. 大規模データにXComを使用 — XComはメタデータ(パス、カウント)用であり、データ自体はS3/GCS経由で受け渡し。XComのpickleは使わない
  3. 誤ったcatchup=True — バックフィルが必要でない限りcatchup=Falseを設定。設定しない場合、Airflowはすべての未実行インターバルを実行します
  4. リトライなし — 一時的な障害はよく発生する。常にretries >= 1と遅延を設定
  5. DAGファイルのトップレベルコード — DAGファイルは30秒ごとに解析される。モジュールレベルでの重いインポートやAPI呼び出しはスケジューラーを遅くします

品質チェックリスト

  • PythonネイティブなDAGにはTaskFlow APIを使用(従来のオペレーターチェーンは使用しない)
  • バックフィルを意図しない限りcatchup=Falseを設定
  • すべてのタスクにリトライを設定(最低1回)
  • クリティカルパスのタスクにSLAを設定
  • XComは参照(S3パス)を渡し、データペイロードは渡さない
  • ファンアウト並列処理に動的タスクマッピングを使用
  • デプロイ前にairflow dags testでDAGをローカルテスト
  • タスクの冪等性: タスクを再実行しても同じ結果が得られる
  • タスク失敗とSLAミスに対するアラート設定
  • Airflow UIでフィルタリングするためのDAGタグを適用

Individual skills in this repo

This repo contains 20 individual skills — each has its own dedicated page.

curiositech/windags-skills

Expert in 2000s-era music visualization (Milkdrop, AVS, Geiss) and modern WebGL implementations. Specializes in Butterchurn integration, Web Audio API AnalyserNode FFT data, GLSL shaders for audio-reactive visuals, and psychedelic generative art. Activate on "Milkdrop", "music visualization", "WebGL visualizer", "Butterchurn", "audio reactive", "FFT visualization", "spectrum analyzer". NOT for simple bar charts/waveforms (use basic canvas), video editing, or non-audio visuals.

curiositech/windags-skills

Expert legal research agent for finding and scraping expungement data state by state. Knows authoritative sources, URL patterns, Firecrawl configuration, and 2026 legal landscape.

curiositech/windags-skills

Expert in 3D computer vision labeling tools, workflows, and AI-assisted annotation for LiDAR, point clouds, and sensor fusion. Covers SAM4D/Point-SAM, human-in-the-loop architectures, and vertical-specific training strategies. Activate on '3D labeling', 'point cloud annotation', 'LiDAR labeling', 'SAM 3D', 'SAM4D', 'sensor fusion annotation', '3D bounding box', 'semantic segmentation point cloud'. NOT for 2D image labeling (use clip-aware-embeddings), general ML training (use ml-engineer), video annotation without 3D (use computer-vision-pipeline), or VLM prompt engineering (use prompt-engineer).

curiositech/windags-skills

Implement WCAG 2.2 AA/AAA compliance with automated testing, keyboard navigation, screen reader support, and focus management. Activate on: accessibility audit, WCAG compliance, keyboard navigation, screen reader, aria attributes, axe-core, focus trap. NOT for: design-level accessibility review (use design-accessibility-auditor), color contrast only (use css-in-js-architect).

curiositech/windags-skills

Time-blind friendly planning, executive function support, and daily structure for ADHD brains. Specializes in realistic time estimation, dopamine-aware task design, and building systems that actually work for neurodivergent minds.

curiositech/windags-skills

Designs digital experiences for ADHD brains using neuroscience research and UX principles. Expert in reducing cognitive load, time blindness solutions, dopamine-driven engagement, and compassionate design patterns. Activate on 'ADHD design', 'cognitive load', 'accessibility', 'neurodivergent UX', 'time blindness', 'dopamine-driven', 'executive function'. NOT for general accessibility (WCAG only), neurotypical UX design, or simple UI styling without ADHD context.

curiositech/windags-skills

>- Apply crisis decision-making research to agent routing, uncertainty triage, and coordination failure analysis in time-pressured systems. Use when diagnosing handoff failures, analytical paralysis, or expert judgment under incomplete information. NOT for routine coding, simple CRUD design, or static single-agent tasks with complete information.

curiositech/windags-skills

Extend and modify the admin dashboard, developer portal, and operations console. Use when adding new admin tabs, metrics, monitoring features, or internal tools. Activates for dashboard development, analytics, user management, and internal tooling.

curiositech/windags-skills

Conversation patterns and interaction protocols for multi-agent systems. Covers request/response, pub/sub, blackboard, delegation chains, debate, critique, consensus, fan-out/fan-in, supervisor-worker, and peer negotiation. Deep analysis of AutoGen conversation patterns, CrewAI delegation, LangGraph state passing, and FIPA-ACL performatives. Teaches how to design what agents say to each other and in what order. Activate on: "agent conversation", "agent protocol", "multi-agent debate", "agent delegation", "supervisor worker pattern", "agent voting", "consensus protocol", "fan-out fan-in", "agent negotiation", "blackboard pattern", "agent dialogue", "conversation topology", "agent handoff". NOT for: wire format or serialization (use agent-interchange-formats), orchestration infrastructure (use agentic-infrastructure-2026), single agent behavior (use agentic-patterns).

curiositech/windags-skills

Meta-agent for creating new custom agents, skills, and MCP integrations. Expert in agent design, MCP development, skill architecture, and rapid prototyping. Activate on 'create agent', 'new skill', 'MCP server', 'custom tool', 'agent design'. NOT for using existing agents (invoke them directly), general coding (use language-specific skills), or infrastructure setup (use deployment-engineer).

curiositech/windags-skills

AI-powered calendar management and agent-based scheduling coordination. Covers calendar APIs (Google Calendar, CalDAV/iCal), AI scheduling assistants (Reclaim, Clockwise, Motion, Cal.com), building custom calendar agents with MCP, multi-calendar merging, timezone management, focus block protection, meeting fatigue detection, and agent-to-agent meeting negotiation protocols. Activate on: "calendar agent", "AI scheduling", "calendar coordination", "meeting scheduling", "calendar API", "focus time protection", "calendar optimization", "Google Calendar MCP", "Reclaim", "Clockwise", "Motion", "Cal.com", "smart scheduling", "calendar-aware agent", "timezone scheduling", "agent negotiation meetings". NOT for: manual calendar UI component design (use form-validation-architect), project management scheduling or Gantt charts (use project-management-guru-adhd), general time-tracking or pomodoro apps (use adhd-daily-planner for time-awareness), building the agent itself from scratch (use agent-creator).

curiositech/windags-skills

Build and adopt production AI agent infrastructure in 2026. Covers framework selection (LangGraph, CrewAI, AutoGen, MCP), orchestration patterns, evaluation, observability, memory systems, and tool use. Also covers the SOCIAL dimension: how to sell agent infrastructure internally, change management, measuring ROI, building trust in autonomous systems, and scaling adoption across teams. Activate on: "agent infrastructure", "agent framework comparison", "which agent framework", "sell AI tools internally", "agent adoption", "agent observability", "agent evaluation", "MCP architecture", "agentic mesh", "enterprise AI agents", "AI change management", "agent ROI". NOT for: building specific agents (use ai-engineer), designing agent behavior patterns (use agentic-patterns), prompt tuning (use prompt-engineer).

curiositech/windags-skills

Fundamental patterns for effective agentic behavior. Teaches decomposition, tool orchestration, error recovery, context management, quality self-assessment, and knowing when to stop. Model-agnostic principles that make any agent more effective regardless of domain. Activate on: "how should I structure this agent", "agentic workflow", "agent patterns", "multi-step task", "tool orchestration", "/agentic-patterns", "decompose this", "agent best practices", "chain of actions", "when should the agent stop", "agent loop design". NOT for: creating agent infrastructure (use agent-creator), building DAGs (use windags-architect), specific tool implementation.

curiositech/windags-skills

Automated discovery and matching of agent skills for dynamic task routing and capability assessment

curiositech/windags-skills

Cryptographic security for agentic systems — zero-trust agent networking, signed message envelopes (JWS/JWE), capability-based security (ocaps), Merkle tree audit trails, WASM sandboxing, and formal verification. Covers CLI dev tool security, mTLS between agents, permission boundaries (least privilege for AI agents), and supply chain security for skills/plugins. Activate on: "agent security", "zero trust agents", "secure agent communication", "capability-based security", "ocap", "signed messages between agents", "agent audit trail", "sandbox agent execution", "agent permissions", "mTLS agents", "cryptographic verification", "agent supply chain", "OWASP agentic", "prove agent did X", "tamper-proof agent logs". NOT for: application-level SAST scanning (use security-auditor), network firewall rules (use infrastructure), SOC2/HIPAA compliance (organizational), or prompt injection defense (use prompt-engineer).

curiositech/windags-skills

Data structures and serialization formats for agent-to-agent communication. Covers message envelopes, structured output schemas, capability declarations, task handoff payloads, error/retry signaling, and context windows as data structures. Deep comparison of A2A protocol, MCP, OpenAI function calling, and LangChain message types. Teaches when to use rigid schemas vs free-form with validation, typed vs untyped, streaming vs batch. Activate on: "agent message format", "agent communication schema", "agent-to-agent protocol", "A2A protocol", "MCP message format", "structured output for agents", "agent interop", "interchange format", "agent serialization", "task handoff format", "capability declaration". NOT for: what agents say to each other (use agent-conversation-protocols), orchestration topology (use multi-agent-coordination), building agent infrastructure (use agentic-infrastructure-2026).

curiositech/windags-skills

Logic-based agent programming language implementing BDI architecture for practical autonomous agent development

curiositech/windags-skills

>- Design AgentSpeak(L)-style BDI agents with context-guarded plans, selection functions, and intention stacks. Use for interruptible autonomy, agent policy, and multi-agent orchestration in dynamic environments. NOT for simple rule engines, static planners, or centralized workflows.

curiositech/windags-skills

Foundational concurrent computation model where actors communicate exclusively through asynchronous message passing

curiositech/windags-skills

license: Apache-2.0 NOT for unrelated tasks outside this domain.

関連スキル