Communitygithub.com

Spark & PySpark 效能調校

撰寫與調校 PySpark 任務——涵蓋連線策略與廣播大小限制、洗牌分割區大小設定、資料傾斜診斷與加鹽處理、UDF 迴避、快取策略以及輸出檔案佈局,並提供具體的大小與傾斜閾值。

Spark & PySpark 效能調校 是什麼?

這項技能專為開發者與 AI 代理設計,用以診斷並最佳化 PySpark 任務的效能。它能協助識別並解決 Spark 任務中最常見的三個瓶頸:可避免的洗牌、傾斜的鍵值以及逐行傳遞的 Python UDF。技能提供一套結構化的操作流程,從收集任務代碼、Spark UI 指標與執行計畫開始,引導使用者設定合理的 SparkSession 組態(例如啟用 AQE 並恰當設定 `spark.sql.shuffle.partitions`),並遵循嚴格的優先順序:優先使用內建 DataFrame API,其次為 Pandas UDF,最後才考慮純 Python UDF。對於連線策略,技能會根據資料表大小決定使用廣播連線(BroadcastHashJoin)或排序合併連線(SortMergeJoin),並提供診斷與處理資料傾斜的具體方法(例如使用 AQE 的傾斜處理或手動加鹽)。此外,技能涵蓋分割區控制(repartition vs coalesce)、輸出檔案大小目標(128 MB – 1 GB)、只在重複使用時才快取以及最終以 df.explain(True) 驗證計畫的正確性。最後的產出包含修訂後的任務程式碼與一份調校備忘錄。此技能適用於 Claude Code、Cursor 等平台,能大幅減少手動除錯的時間。

相容平台~Claude Code~Codex CLI~Cursor
npx skills add https://github.com/SkillMedev/skills/tree/HEAD/skills/spark-jobs

在你喜歡的 AI 中提問

開啟一個已預先載入此 Agent Skill 的新對話。

說明文件

Spark & PySpark

幾乎每個緩慢的 Spark 任務都離不開三個原因之一:可避免的洗牌、傾斜的鍵值,或是逐行透過 Python 傳遞資料。最常見的錯誤是在讀取執行計畫之前就先調整叢集規模——花更多錢買更多執行器來執行同樣浪費的 DAG,只是跑得更快而已。

操作流程

步驟 1:收集輸入

  1. 任務的程式碼;如果是緩慢的任務,則需要 Spark UI 中該階段的檢視(任務持續時間分佈、洗牌讀取/寫入大小)以及 df.explain(True) 的輸出。
  2. 輸入資料的大小與格式,以及每個連線兩側資料表的大小。
  3. 叢集配置:執行器數量、核心數與記憶體。
  4. 任務是新撰寫還是調校既有任務——調校時,先在動程式碼之前從執行計畫進行診斷。

步驟 2:從合理的會話開始

from pyspark.sql import SparkSession
from pyspark.sql import functions as F

spark = (
    SparkSession.builder
    .appName("etl")
    .config("spark.sql.shuffle.partitions", "200")
    .config("spark.sql.adaptive.enabled", "true")
    .getOrCreate()
)

啟用自適應查詢執行 (AQE);它會在執行階段合併洗牌分割區並切換連線策略。然後審慎設定 spark.sql.shuffle.partitions:目標是每個洗牌分割區 100-200 MB,因此分割區數 ≈ 洗牌資料大小 ÷ 128 MB,且至少要是總執行器核心數的 2-3 倍,以確保沒有核心閒置。

步驟 3:停留在 DataFrame API 中,遠離 Python

DataFrame 操作會透過 Catalyst 和 Tungsten 進行,獲得查詢最佳化與堆外記憶體的好處。只有在沒有 DataFrame 原生功能可用時,才降級使用 RDD。Python UDF 會將每一行序列化到 Python 工作者再序列化回來,破壞程式碼生成。優先順序如下:

  1. 內建的 pyspark.sql.functions(例如 F.regexp_extract、F.when)。
  2. Pandas(向量化)UDF,當 Python 無法避免時使用。
  3. 純 Python UDF 只作為最後的手段。
from pyspark.sql.functions import pandas_udf

@pandas_udf("double")
def normalize(s):
    return (s - s.mean()) / s.std()

步驟 4:根據大小選擇連線策略

  • 當較小的一側能放進記憶體時,使用廣播連線。Spark 會自動廣播低於 spark.sql.autoBroadcastJoinThreshold(預設 10 MB)的資料表;當執行器記憶體允許時,可以明確廣播數百 MB 的資料表,並將約 1 GB 視為實務上限——廣播會將資料表複製到每個執行器,判斷錯誤會導致任務 OOM。
from pyspark.sql.functions import broadcast
result = large.join(broadcast(small), "key")
  • 在連線前先過濾與投影欄位,以縮小洗牌量;每個被移除的欄位與列都是節省的洗牌位元組。
  • 兩張大表 → 接受排序合併連線,但先檢查傾斜(步驟 5)。

步驟 5:診斷並修正傾斜

傾斜的症狀:階段已完成,只剩一兩個任務;最大任務持續時間約為中位數的 3-5 倍以上;Spark UI 中某個洗牌讀取分割區遠大於其他分割區。AQE 的傾斜連線處理會在分割區同時大於中位數分割區大小的 5 倍且超過 256 MB(預設值)時拆分割區——請在執行計畫中確認它是否觸發。當 AQE 無法解決時(例如傾斜的聚合),對熱鍵進行加鹽:在大側的鍵上附加隨機後綴 0..N,將小側在所有 N 個後綴上展開,進行連線,然後移除鹽值。選擇 N ≈ 傾斜因子(某個鍵是中位數的 20 倍時,取 N=20)。

步驟 6:控制分割區與輸出佈局

  • repartition(n, col) 會進行完整的洗牌以平衡資料;在進行大範圍寫入前使用。
  • coalesce(n) 減少分割區數量而不引發洗牌;用於避免產生過小的輸出檔案。
  • 目標輸出檔案大小為 128 MB - 1 GB;數千個小檔案會懲罰每個下游讀取者。
  • 只根據低基數欄位(日期、地區——幾十到幾百個值,絕不能是 ID)進行分割區輸出:
df.write.partitionBy("dt").mode("overwrite").parquet("/data/out")

步驟 7:只快取重複使用的資料

只有在 DataFrame 在 DAG 中被多次使用時才進行快取,具體化它,並在使用完畢後取消快取:

df.cache()
df.count()  # 具體化
# ... 重複使用 df ...
df.unpersist()

快取只使用一次的 DataFrame 浪費記憶體,並可能逐出原本重要的資料。

步驟 8:用執行計畫驗證

重新執行 df.explain(True) 並確認預期的策略出現(BroadcastHashJoin、推送的過濾條件、沒有 Exchange 出現則表示已被消除)。讀取欄位式格式(Parquet、ORC)以使謂詞下推生效,絕對不要對大型資料執行 collect() 到驅動程式,並且在鏈式調用中優先使用 F.col 引用而非字串欄位。

實際範例:錯誤 vs 正確的連線

錯誤——一個 2 TB 的事件資料表連接到一個 200 MB 的維度表,全寬度,使用 Python UDF 進行解析:

result = (events.join(dims, "store_id")          # sort-merge:洗牌所有 2 TB
    .withColumn("region", parse_region_udf("meta"))  # 逐行 Python
    .filter(F.col("dt") == "2024-06-01"))            # 在連線之後才過濾

執行計畫顯示兩側都有 Exchange(2 TB 被洗牌),且 UDF 阻斷了程式碼生成。

正確——先過濾與投影,廣播維度表,使用內建函數:

result = (events
    .filter(F.col("dt") == "2024-06-01")              # 先修剪到約 30 GB
    .select("store_id", "amount", "meta")
    .join(broadcast(dims.select("store_id", "region_code")), "store_id")
    .withColumn("region", F.regexp_extract("meta", r"region=(\w+)", 1)))

現在執行計畫顯示 BroadcastHashJoin,且大側沒有 Exchange:約掃描 30 GB,大表零洗牌,沒有 Python 邊界。

交付產出

產出修訂後的任務程式碼,加上一份調校備忘錄,內容包含:每個連線的連線策略及大小證據、洗牌分割區設定及其計算依據、發現的任何傾斜(最大 vs 中位數任務比率)及採用的修正措施、快取決策、輸出檔案大小預期,以及調校前後的 explain 差異。

不可做的事

  • 絕對不要在沒有估算大小的情況下廣播資料表——誤判的廣播會同時讓所有執行器 OOM。
  • 在檢查 pyspark.sql.functions 之前,絕對不要撰寫 Python UDF;幾乎所有字串、日期與條件邏輯都有內建函數可用。
  • 絕對不要對大型 DataFrame 執行 collect() 或 toPandas();這會將叢集的所有資料經由一個驅動程式傳遞。
  • 絕對不要根據高基數欄位進行分割區輸出——會產生數百萬個目錄,每個目錄只有一行。
  • 絕對不要為了某個因傾斜而變慢的階段而增加執行器;更多核心無法拆開一個分割區。
  • 絕對不要反射性地快取;只快取在後續步驟中重複使用的 DataFrame,並在使用後取消快取。

品質標準

完成的任務只有在滿足以下條件時才算通過:每個連線都標明其策略與大小證據;spark.sql.shuffle.partitions 是根據資料大小推導而來,而非保留習慣值;Spark UI 中洗牌階段的任務持續時間沒有超過中位數的約 3 倍(或傾斜已說明並修正);沒有純 Python UDF 在存在內建函數的情況下存活;輸出檔案落在 128 MB - 1 GB 的範圍內;且最終的 explain 確認了預期的執行計畫。

Individual skills in this repo

This repo contains 19 individual skills — each has its own dedicated page.

SkillMedev/skills

Designs REST API surfaces - resource naming, HTTP method and status-code semantics, error shapes, pagination, and filtering - and delivers an endpoint spec a consumer can build against without asking questions. Use when someone asks "how should I name this endpoint", "what status code should this return", "should this be PUT or PATCH", "how do I paginate this list", or is reviewing an API before it ships to external consumers. Do NOT use for planning breaking-change rollouts and deprecation windows - use api-versioning-strategist instead; for GraphQL type and resolver design - use graphql-schema instead; for generating client SDKs from an existing spec - use api-client-generator instead; for designing inbound webhook endpoints - use webhook-receiver-hardener instead.

SkillMedev/skills

Produce an accurate, properly timed caption track (SRT or WebVTT) from a video's audio - transcribing or aligning to the voiceover script, timing cues to speech, and enforcing line-length and reading-speed rules so captions are readable and in sync. Use when someone says "generate captions", "make subtitles from the audio", "transcribe and caption this", "export an SRT or VTT", "the captions are out of sync", or "the subtitles flash by too fast to read". Do NOT use to burn-in, style, reframe, or position an existing caption track for a platform (9:16, brand styling, safe areas) - that is social-video-formatter; do NOT use to animate text word-by-word as a motion graphic - that is kinetic-typography; do NOT use to write the spoken script in the first place - that is narration-script.

SkillMedev/skills

Turns data and charts into a decision-driving narrative structured as headline finding, trend, implication, and recommended action - with finding-led chart titles, context for every number, annotation guidance, and honest flags on any conclusion the data cannot support. Use when someone says "turn these numbers into a story", "what's the takeaway from this data", "help me present these results to leadership", or has charts but no narrative. Do NOT use for compressing a long document into a one-pager - use executive-summary instead - or for running the analysis that produces the findings - use eda-playbook instead.

SkillMedev/skills

Use when a task needs live or historical money data - "convert USD to EUR", "current/past exchange rate", "FX rate on this date / over this range", or "current price of Bitcoin/Ethereum, market cap, 24h change". Frankfurter (ECB reference rates, no key) is the FX default; CoinGecko's free keyless tier covers crypto. Do NOT use for stock quotes or equities - no keyless stock API survives verification, say so instead of guessing; do NOT use for country economic indicators like GDP or inflation series - use government-open-data instead; if the request is a vague "I need live data", route through public-data-api-picker.

SkillMedev/skills

Builds a driver-based FP&A operating model linking business inputs to P&L, balance sheet, and cash flow outputs. Use when building an annual plan, preparing investor materials, running scenario analysis, or stress-testing the business.

SkillMedev/skills

Use when a task needs live geographic lookups - "geocode this address", "what's at these coordinates" (reverse geocoding), "lat/lon for this city", "which country/state is this ZIP or postal code in", or "country facts: capital, currency, population, flag". Nominatim (OpenStreetMap) is the geocoding default; Zippopotam for postal codes; APICountries for country facts. All keyless. Do NOT use for weather at a location - use weather-climate instead; do NOT use for country-level statistics over time (GDP, population trends) - use government-open-data instead; if the request is a vague "I need live data", route through public-data-api-picker.

SkillMedev/skills

Runs the full Getting Things Done loop - capture, clarify, organize, reflect, engage - building a trusted system of context lists, a projects list with defined next actions, and a weekly review habit. Use when someone says "I'm overwhelmed and things are slipping through the cracks", "set up GTD for me", "help me do a brain dump and organize it", or "my to-do list is a mess". Do NOT use for just running the weekly review ritual itself - use weekly-review instead - or for clearing an email backlog - use inbox-zero.

SkillMedev/skills

Processes any email backlog to zero using the 4Ds - Delete, Delegate, Defer, Do - with a mass-archive strategy for the obvious, a touch-each-email-once discipline, and a keep-it-clear system of batched processing windows, ruthless unsubscribing, filters, and a minimal folder setup. Use when someone says "I have 5,000 unread emails", "help me get to inbox zero", "email is eating my whole day", or treats their inbox as a to-do list. Do NOT use for drafting the reply emails themselves or prioritization rules for an ongoing support queue - use email-triage instead - or for protecting focus time around the email windows - use deep-work-planner instead.

SkillMedev/skills

Write landing-page copy section by section - hero, social proof, features-as-benefits, objections, CTA - leading with one promise and proving it down the page. Use when someone asks "write my landing page", "improve my hero headline", "what should the homepage say", "turn these features into benefits", or is shipping a new page for a launch or campaign. Do NOT use for testing and optimizing an existing page's conversion rate - use landing-page-cro instead; for the one-line claim itself use positioning-statement; for the full message ladder use messaging-hierarchy.

SkillMedev/skills

Improves an existing landing page's conversion rate through evidence-first CRO - audit heatmaps and recordings, write structured hypotheses, prioritize with ICE scoring, and run one-variable tests to statistical significance. Use when someone asks "why isn't my landing page converting", "audit this page before we scale spend", "which A/B test should we run first", or after a CVR drop or ahead of a paid-traffic push. Do NOT use for writing a new landing page from scratch - use landing-page-copy instead. Do NOT use for auditing the ad accounts sending the traffic - use paid-acquisition-audit instead. Do NOT use for reading the results of a finished experiment - use ab-test-analyzer instead.

SkillMedev/skills

Runs structured coaching sessions using values clarification and the GROW model, ending every session with one committed action, a deadline, and an if-then plan for the likely obstacle. Use when someone says "I feel stuck in my life", "help me figure out what I want", "hold me accountable to my goals", or "coach me through this decision". Do NOT use for building a stress toolkit - use stress-management instead - or a journaling practice - use journal-framework; for a standing goal-tracking system, use goals-accountability. Coaching, not therapy: signs of clinical distress route to a licensed professional.

SkillMedev/skills

Adapts English content for a target language and region - swapping idioms, setting the right formality register, replacing cultural references, converting dates, units, and currency - and delivers the localized text with a notes column explaining each adaptation and flagging what needs a native reviewer. Use when someone asks "localize this copy for the Japanese market", "adapt our marketing copy for the Mexican market", "make this translation sound native instead of literal", or a literal translation reads foreign, stiff, or risky. Do NOT use for auditing in-product UI text quality - use ux-writing-audit instead.

SkillMedev/skills

Turns a natural-language video brief into a complete, ready-to-preview Remotion composition - extracts duration, scenes, brand colors, aspect ratio, and real copy; plans the frame budget; and writes data-driven React/TypeScript using useCurrentFrame, interpolate, spring, AbsoluteFill, and Sequence, registered in Root.tsx. Use when someone says "build a 20-second product demo video", "animate a feature walkthrough", "write the Remotion code for this marketing clip", or wants a scene-by-scene composition they can scrub in Studio. Do NOT use for rendering, iterating, or batching the finished MP4 - use remotion-render instead - or for installing and scaffolding the project - use remotion-setup instead.

SkillMedev/skills

Scaffolds a new Remotion video project wired for Claude Code Agent Skills - Node check, create-video scaffold, skills install, folder conventions, Google Fonts, and a smoke-test render. Use whenever someone wants to start making videos with Remotion and Claude, even just "make product videos with Claude.

SkillMedev/skills

Classifies incident severity (SEV1-4) using impact, scope, and urgency signals and decides who to page. Use when an alert fires or a report comes in and a severity call must be made quickly.

SkillMedev/skills

Use the Skill Me catalog from inside any conversation - discover, install, and manage Claude skills through the Skill Me MCP, and load installed skills automatically each session.

SkillMedev/skills

Write one platform-native caption from a topic and brand voice, with a front-loaded hook, native length, and a single earned CTA. Use when the user gives a topic (and optionally brand voice or a described image/video) and asks for an Instagram, LinkedIn, X/Twitter, or TikTok caption. Do NOT use when the user wants a dedicated standalone LinkedIn post - use linkedin-post-writer instead; or a multi-post X thread - use tweet-thread-builder instead; or to adapt one existing piece of content into versions for several channels - use cross-platform-reformatter instead.

SkillMedev/skills

Reframe and export a finished video for social platforms - 9:16 ↔ 16:9 cropping, burned-in captions, a platform-native first frame, loop design, and per-platform specs (aspect, length, safe areas) for TikTok, Reels, Shorts, X, and LinkedIn. Use when someone says "make this vertical", "crop to 9:16", "format for TikTok/Reels/Shorts", "add captions/subtitles", "why is my video cut off on mobile", "resize for Instagram", or "export for social". Do NOT use when the question is about the storyboard or shot order before any video exists (use video-storyboard); the animation craft of how elements move (use motion-design-principles); animated word-by-word caption styling (use kinetic-typography); palette, contrast, or lighting (use motion-color-and-light); directing the product-demo content and screen choreography (use product-demo-director); or scoring, beat-syncing, or audio (use sound-and-music-sync) - this is the craft layer that decides WHAT to format for the feed, then hands a concrete spec to remotion-compo...

SkillMedev/skills

Builds clean, performant, accessible SwiftUI views with correct state ownership, scoped invalidation, and smooth list scrolling, and reviews existing SwiftUI code against a concrete frame-time and re-render budget. Use when someone asks "why does my SwiftUI list stutter", "should this be @State or @Observable", "my whole screen re-renders when one row changes", "how do I animate this transition", or wants a SwiftUI view built or refactored. Do NOT use for cross-platform React Native apps - use react-native-pro instead; do NOT use for Flutter widget trees - use flutter-widget-architect instead; do NOT use for Android Compose UIs - use jetpack-compose-builder instead.

相關技能