Communitygithub.com

G1Joshi/Agent-Skills

專家級的 Dask 分散式運算協助,涵蓋 Dask DataFrames、Arrays、Futures 以及叢集擴展。適用於分析單機或多節點叢集中超出 pandas 處理能力的龐大資料集。

Agent-Skills 是什麼?

此技能為 AI 代理提供完整的 Dask 分散式運算支援,專為處理超過單機記憶體容量的資料集設計。它涵蓋 Dask DataFrame(用於超大表格資料)、Dask Array(用於分散式線性代數)、dask.delayed(用於自訂任務圖形)以及 dask.distributed(用於多節點叢集管理)等核心功能。技能內容包括實際的程式碼範例,說明如何惰性載入大型 Parquet 檔案、使用工作流程圖平行化 API 呼叫,以及操控超大規模的多維陣列。它還提供了 2026 年的最佳實務(例如建議的分區大小、持續化中間結果以及避免在迴圈過早調用 .compute()),並包含常見錯誤(如 KilledWorker 和物件傳送過大)的故障排除指南。此技能支援所有使用 Python 的 Dask 版本,並特別強調高效能 Shuffle 和 GPU 整合,使其成為處理大規模資料管線的開發人員與資料科學家的關鍵工具。

相容平台~Claude Code~Codex CLI~Cursor
npx skills add https://github.com/G1Joshi/Agent-Skills/tree/HEAD/skills/ai-ml/dask

在你喜歡的 AI 中提問

開啟一個已預先載入此 Agent Skill 的新對話。

說明文件

Dask

Dask 可擴展 Python。它看起來像 Pandas/NumPy,卻能在叢集上執行。2025 年的更新重點在於高效能 Shuffle 與 GPU 整合。

使用時機

  • 超出記憶體容量的表格計算:在單機或多節點叢集中處理 10GB-1TB 的 CSV/Parquet 資料集。
  • 平行化自訂 Python 工作流程:使用 dask.delayed 來平行化任意的 Python 迴圈與任務圖形。
  • 分散式機器學習:與 Scikit-learn、XGBoost 和 LightGBM 整合,並跨分散式工作者節點執行。
  • NumPy 與 Pandas 擴展:以熟悉的 API 進行超出核心記憶體的陣列與 DataFrame 計算。

快速入門

import dask.dataframe as dd

# 惰性讀取數 GB 的分割 CSV 檔案
df = dd.read_csv('data/transactions_*.csv')

# 跨所有 CPU 核心平行計算彙總統計值
result = df.groupby('category').amount.mean().compute()
print(result)

核心概念

#使用 Dask DataFrame 進行超出核心記憶體處理

惰性載入分割的 Parquet 檔案並計算彙總值:

import dask.dataframe as dd
from dask.distributed import Client

# 初始化本地分散式叢集
client = Client(n_workers=4, threads_per_worker=2, memory_limit='4GB')
print(f"Dask 儀表板位置:{client.dashboard_link}")

# 惰性載入多檔案資料集
df = dd.read_parquet(
    's3://analytics-bucket/transactions/*.parquet',
    columns=['customer_id', 'amount', 'status', 'country'],
    engine='pyarrow'
)

# 惰性建立計算圖形
filtered = df[df['status'] == 'completed']
metrics = filtered.groupby('country')['amount'].agg(['mean', 'sum', 'count'])

# 執行圖形並回傳具體的 Pandas DataFrame
result_df = metrics.compute()
print(result_df)

#使用 dask.delayed 建立自訂任務圖形

平行化獨立的函式執行:

from dask import delayed, compute
import time

@delayed
def fetch_api_data(endpoint: str) -> dict:
    time.sleep(0.5) # 模擬 I/O
    return {'endpoint': endpoint, 'status': 200, 'items': [1, 2, 3]}

@delayed
def transform_payload(data: dict) -> int:
    return sum(data['items'])

@delayed
def aggregate_totals(totals: list[int]) -> int:
    return sum(totals)

# 建構依賴關係 DAG
endpoints = ['users', 'orders', 'inventory', 'payments']
fetched = [fetch_api_data(ep) for ep in endpoints]
transformed = [transform_payload(f) for f in fetched]
final_total = aggregate_totals(transformed)

# 跨叢集平行執行所有任務
total_result = final_total.compute()
print("總計:", total_result)

#Dask Array:分散式線性代數

操作超大型多維陣列:

import dask.array as da

# 建立一個 50,000 x 50,000 的陣列,分割成 5,000 x 5,000 的區塊
x = da.random.normal(10, 0.1, size=(50000, 50000), chunks=(5000, 5000))

# 惰性執行計算
y = x + x.T
z = y[::2, ::2].mean(axis=0)

# 計算結果
result = z.compute()
print("計算後的陣列平均值形狀:", result.shape)

常見模式

針對不規則平行任務圖形的分散式 Futures

問題:無法整齊對應到 DataFrame/Array 行列結構的自訂平行計算。

解決方案: 使用 Dask Distributed Client 搭配非同步 futures:

from dask.distributed import Client

client = Client(n_workers=4, threads_per_worker=2)

def train_partition(part_id: int):
    # 訓練局部子模型
    return f"模型 {part_id} 已訓練"

futures = [client.submit(train_partition, i) for i in range(10)]
results = client.gather(futures)
print(results)

最佳實務 (2026)

  • 務必檢查 Dask Web 儀表板(通常位於 port 8787)以監控記憶體壓力、任務串流與瓶頸。
  • 務必在記憶體中選擇介於 100MB 到 300MB 的區塊大小,以達到最佳平行效率。
  • 務必在重複查詢相同轉換後的資料時,持續化(persist)中間 DataFrame(df = df.persist())。
  • 務必及早使用欄位投影與謂詞下推來過濾欄位與列。
  • 切勿在迴圈內呼叫 .compute();應建立完整的任務圖形後再呼叫一次 compute()。
  • 切勿在資料能舒適地放入 RAM 時使用 Dask;原生 Pandas 與 Polars 在記憶體任務上明顯更快。
  • 切勿建立數百萬個微小的延遲(delayed)任務;過多的任務開銷會降低排程器效能。

故障排除

錯誤原因解決方案
KilledWorker: Worker exceeded 95% memory budget任務分區太大,無法在計算時放入工作者 RAM。增加分區數量:df.repartition(npartitions=100)。
UserWarning: Sending large object to workers將大型 NumPy 陣列或 DataFrame 直接作為函式參數傳遞。在提交任務前先用 client.scatter(data) 將資料分散一次。
Slow compute: Too many tiny tasks數千個微型分區導致任務排程開銷過高。使用 repartition(partition_size="100MB") 合併小分區。

參考資料

Individual skills in this repo

This repo contains 11 individual skills — each has its own dedicated page.

G1Joshi/Agent-Skills

Expert dbt (data build tool) assistance covering SQL modeling, Jinja macros, tests, documentation, and semantic layer. Use when building analytics engineering pipelines on BigQuery, Snowflake, or PostgreSQL.

G1Joshi/Agent-Skills

Expert JAX assistance covering Autograd, XLA compilation (`jit`), vectorization (`vmap`), and parallelization (`pmap`). Use when building high-performance numerical computing and cutting-edge deep learning research.

G1Joshi/Agent-Skills

Expert Ray distributed computing assistance covering Ray Core (actors, tasks), Ray Train, Ray Tune, and Ray Serve. Use when scaling Python compute and ML training across multi-node clusters.

G1Joshi/Agent-Skills

Expert Git version control assistance covering branching, interactive rebase, cherry-pick, submodules, worktrees, and conflict resolution. Use when managing source code history and collaboration workflows.

G1Joshi/Agent-Skills

Expert K9s CLI assistance covering terminal Kubernetes cluster navigation, real-time log streaming, port forwarding, and pod debugging. Use when managing and troubleshooting Kubernetes clusters with speed.

G1Joshi/Agent-Skills

Expert SWC assistance covering high-performance Rust-based JavaScript/TypeScript compilation, .swcrc configuration, Jest testing via @swc/jest, and minification. Use when replacing Babel with SWC for faster builds, accelerating test execution, or compiling modern ECMAScript features.

G1Joshi/Agent-Skills

Expert Tig assistance covering text-mode interface for Git, interactive staging, commit graph visualization, diff exploration, and blame navigation. Use when navigating Git commit history in the terminal, staging hunks interactively, browsing file changes, or reviewing revisions.

G1Joshi/Agent-Skills

Expert Vim assistance covering modal editing, .vimrc configuration, registers, macros, search/replace, and plugin management via vim-plug. Use when editing text efficiently in terminal environments, writing Vimscript, recording macros, or configuring core Vim settings.

G1Joshi/Agent-Skills

Expert Zed editor assistance covering high-performance Rust-based text editing, multi-buffer editing, language server protocols (LSP), and AI assistant integrations. Use when configuring Zed settings.json, setting up language extensions, collaborating in real-time channels, or optimizing editor startup speed.

G1Joshi/Agent-Skills

Expert Zsh assistance covering shell customization, Oh My Zsh plugins, Zinit plugin manager, prompt engineering (Starship/Powerlevel10k), and shell scripting. Use when configuring .zshrc, writing Zsh automation scripts, optimizing shell startup time, or configuring tab-completion.

G1Joshi/Agent-Skills

Expert [skill-name] assistance covering [feature 1], [feature 2], and [feature 3]. Use when [working with X], [debugging Y], or [implementing Z].

相關技能