Communitygithub.com

OneWave-AI/claude-skills

建立並執行 System One 模型的標籤化評估集,對比提示詞(criteria)措辭與閾值,找出最佳配置。適用於 Jev/Von 分類不準確或不可靠時、在託管 API 與本地模型間做選擇、調整 noul 閾值,或在上線任何 typed-decision 功能前使用。產出「措辭 vs. 準確度」矩陣與校準後的閾值。

claude-skills 是什麼?

此技能提供一套嚴謹的評估工作流程,用於評估和調校 System One 模型(特別是 Jev 和 Von 這類 typed-decision 模型)的表現。核心方法論是:模型精確度深受提示詞(criteria)措辭影響,且失敗模式是靜默的(模型會自信地給出類型正確但答案錯誤的結果),因此建立標籤化的真實評估資料集是首要任務。技能引導使用者建立至少 50 筆(上線前建議 200+)從實際流量取樣的真實資料,然後通過撰寫多組(例如簡潔、詳細、含預設值)不同的 criteria 配置來進行「措辭掃描」,最後對每個 noul(無閾值選項)進行閾值掃描,並誠實地評估信心門檻(confidence gate)的效果。它支援 Jev、Von 以及任何 typed-decision 配置,並能在所有支援的平台上執行。最終產出一個包含各配置/模型準確率、選定閾值、門檻誤差攔截率與流量提升量的完整報告,並明確說明評估集的缺點,避免過度解讀小樣本結果。

相容平台✓Claude Code~Codex CLI~Cursor
npx skills add https://github.com/OneWave-AI/claude-skills/tree/HEAD/jev-eval

在你喜歡的 AI 中提問

開啟一個已預先載入此 Agent Skill 的新對話。

說明文件

評估 typed-decision 配置

評估集本身就是產品。 System One 模型的準確度很大程度上取決於問題的撰寫方式,而其失敗模式是靜默的 —— 它會回傳一個自信、類型有效但錯誤的答案。沒有標籤,你就無法區分是問題寫得不好,還是模型本身有問題。

實際案例:重新撰寫 criteria 讓一個開放模型在 15 筆記錄上的成績從 4/15 提升到 14/15,完全沒有更動模型。接著在 150 筆記錄上進行同樣的比較,該模型整體準確率為 61%,而 Jev 則是 97% —— 15 筆記錄的讀數只是因為樣本小且題目簡單所造成的假象。這兩個事實的意義在於:措辭會影響結果,且小樣本會誤導方向。

執行

python ~/.claude/skills/jev-eval/scripts/sweep.py labelled.json configs.json \
    --backend jev|von --question <name>

labelled.json 的格式為 [{"id","state","truth"}]。configs.json 將配置名稱對應到 {"instructions", "criteria"} —— 如果 criteria 是個 dict,代表選項,是為 choice;如果是個 list 代表等級,是為 score;如果省略,則表示是 noul。腳本會從鑰匙圈(typesafe-api-key)讀取 Jev 金鑰,印出每個配置的準確率,將離散程度標記為 ROBUST 或 FRAGILE,對 noul 進行閾值掃描,並評估信心門檻。

真實輸出範例:針對 50 筆 agent shell 指令風險記錄,僅更換後端模型:

config                         accuracy   ms/rec
A terse one-liners               45/50       410      <- Jev
B rich criteria                  49/50       415
C rich + exclusions              46/50       418
D deliberately lazy              44/50       411
spread: 44/50 to 49/50   (ROBUST - wording is not load-bearing)

A terse one-liners                9/50        66      <- Von, same configs
B rich criteria                  23/50       121
C rich + exclusions              15/50       129
D deliberately lazy              22/50        53
spread: 9/50 to 23/50    (FRAGILE - and the ceiling is still not usable)

先看天花板,再看離散度。 一個 FRAGILE 模型,即使最佳配置也只有 23/50,這不是靠修改措辭就能解決的問題 —— 這是選錯了模型。

1. 建立評估集

最少 50 筆記錄,上線前建議 200 筆以上。 從真實流量中取樣,不要使用合成資料。

  • 包含平凡的中間地帶,不只是乾淨的案例和戲劇性的邊界情況
  • 包含中繼資料遺失或損毀的記錄 —— 分類器往往在這裡失敗
  • 在執行任何模型之前,透過閱讀記錄來進行標籤。永遠不要根據模型輸出做標籤
  • 儲存為 JSON,包含原始記錄與 truth 欄位
[{"id":"D-1994","state":"...完整記錄文字...","truth":"inbound_prospect"}]

如果你自己都無法自信地標籤一條記錄,那模型也做不到 —— 要嘛放棄這條記錄,要嘛修正問題讓答案變得明確。

2. 掃描措辭,不只是模型

這是核心步驟。撰寫 3–4 個本質上不同的 criteria 配置,然後全部執行:

  • A —— 每個選項只用一行簡潔描述(大多數人最初寫的版本)
  • B —— 每個選項用 3–4 句話加上具體範例
  • C —— B 的基礎上加上明確的預設值和明確的排除條件("只有在…時才")
  • D —— 故意懶惰,只用四到五個字,作為下限測試

報告每個配置在每個模型上的準確率:

config                    JEV       VON      LAYA     (lead triage, n=50)
A terse one-liners      47/50     34/50     21/50
B rich criteria         49/50     28/50     15/50
C rich + exclusions     48/50     24/50     19/50
D deliberately lazy     48/50     22/50     24/50

先看下限,再看離散度。 下限是「如果我寫得很糟,這個模型還能勝任工作嗎?」;離散度是「維護這個配置會花多少成本?」。根據指令任務的實際測量,每個託管模型的下限都在 82–92%(Haiku 46-48, GPT-4.1-mini 45-50, Jev 44-49, GPT-5-mini 41-49),而 Von 的下限是 9/50,Laya 是 18/50。請注意,Jev 並不像小 LLM 那樣對措辭更穩健 —— 它同樣有約十個百分點的波動。你買到的是下限,而不是免疫力。

請注意,左欄的結果並沒有呈現一個清晰的「越詳細越好」的趨勢。Von 在此的最佳配置反而是簡潔版。任何能夠移動開放模型數字的能力,反映的是對表面形式的敏感度,而不是真正的理解,所以不要假設下一次重寫 criteria 就一定能提升表現 —— 用評估集重新跑一遍。

3. 為每個 noul 掃描閾值

永遠不要直接使用 0.5。掃描並閱讀曲線:

for t in [0.5,0.6,0.7,0.75,0.8,0.85,0.9,0.95]:
    tp = sum(p>=t and y     for p,y in z); fp = sum(p>=t and not y for p,y in z)
    fn = sum(p< t and y     for p,y in z); tn = sum(p< t and not y for p,y in z)
    print(f"{t}  P={tp/max(tp+fp,1):.2f}  R={tp/max(tp+fn,1):.2f}  acc={(tp+tn)/len(z):.2f}")

不同模型有不同的下限。Jev 的 noul 在那些明顯乾淨的記錄上落在 0.2–0.5,而 Claude 則降到 0.0 —— 所以 Jev 的真正閾值大約是 0.85。為某個模型調校的閾值無法轉移到另一個模型。更換模型時請重新掃描。

4. 誠實評估信心門檻

兩個數字,永遠一起看:

errs   = [r for r in rows if r.pred != r.truth]
rights = [r for r in rows if r.pred == r.truth]
for g in [0.5,0.7,0.9]:
    caught    = sum(r.conf <  g for r in errs)      # 門檻攔截到的錯誤
    escalated = sum(r.conf <  g for r in rows)      # 被提升的總流量
    print(f"gate {g}: catches {caught}/{len(errs)} errors, escalates {escalated/len(rows):.0%} of volume")

一個能攔截 10/11 個錯誤,但卻提升了 93% 流量的門檻,不是一個有效的門檻 —— 它只是一個多了額外步驟的慢路徑。沒有良好準確度的校準,買不到任何東西。

5. 報告

  • 每個配置、每個模型的準確率,並標明離散程度
  • 為每個 noul 選擇的閾值,以及支撐該選擇的掃描結果
  • 門檻:攔截的錯誤數 以及 提升的流量比例
  • 在生產量下,預估的延遲與每千次請求成本
  • 明確說明評估集的大小以及未覆蓋的範圍

小樣本會誤導。如果兩個模型在 15 筆記錄上都得到 100%,這無法區分任何東西 —— 直接說出來,而不是暗示這個平手有意義。

相關技能

jev-integrate 是這個流程對接的整合工作流程。jev-audit 則用於尋找值得評估的候選案例。

Individual skills in this repo

This repo contains 20 individual skills — each has its own dedicated page.

OneWave-AI/claude-skills

Deploy a 2-layer parallel agent hierarchy for large, parallelizable work — big refactors, multi-file migrations, codebase-wide audits, bulk generation. A top-tier commander (Fable or Opus) orchestrates the swarms; the user picks a power level (Max Power / Heavy / Balanced / Economy) that sets the Opus/Sonnet/Haiku model mix per layer. Layer 1 is 3-50+ specialist agents, each with its own full context window; Layer 2 is 2+ sub-agents per member. Includes git safety, tiered sizing, a pre-deploy gate, phantom-completion checks, and multi-wave follow-up.

OneWave-AI/claude-skills

Generate animated videos and motion graphics from natural language descriptions. Creates a standalone Vite + React project with Framer Motion scenes that auto-play in the browser. Use when the user wants to create animations, motion graphics, video intros, animated presentations, or product demos.

OneWave-AI/claude-skills

Use before starting a product, feature, MVP, agent, automation, or integration build to identify the main pre-build risk and choose the smallest validation step. Trigger when the user asks whether to build, scope, validate, prioritize, or sanity-check an idea before implementation.

OneWave-AI/claude-skills

Post-mortem analysis when a client churns. Takes client history, engagement data, support tickets, usage logs, and exit feedback to produce a comprehensive churn autopsy with root cause classification, timeline of decline, and preventive measures.

OneWave-AI/claude-skills

Compose a premium, animated landing page section by section in Next.js + Tailwind + Motion (Framer Motion) — built on a real design system, with editorial copy and motion that feels human-crafted, not template-generated. Use when building or rebuilding a landing page, marketing site, or hero/feature/pricing/CTA sections that need to look production-ready and bespoke from day one.

OneWave-AI/claude-skills

Assemble 2-3 complementary experts to collaboratively analyze anything. Experts work together to explore topics from multiple expert angles.

OneWave-AI/claude-skills

Convert any topic into playable browser games. Types: trivia, matching, word puzzles, adventure games. Uses Phaser.js or Kaboom.js.

OneWave-AI/claude-skills

Turn a marketing brief or product offer into a finished short-form video ad built in HyperFrames — hook, script, shot-by-shot storyboard, scene-by-scene HTML composition, captions, and voiceover. Outputs platform cuts (vertical 9:16, square, 16:9). Use when you need a launch ad, product promo, social video, or paid-creative concept ready to render.

OneWave-AI/claude-skills

Turn a website URL, product docs, or a founder's rambling description into a tight 60-90 second explainer video built in HyperFrames -- problem/solution narrative, animated product walkthrough scenes, captions, and voiceover, ready for the homepage hero, sales emails, and onboarding.

OneWave-AI/claude-skills

Build promo videos for local businesses in HyperFrames -- the new-menu drop, the seasonal offer, the event announcement, the "we're hiring" post, the Google Business Profile clip. Restaurant, salon, gym, trades: 15-30 seconds, brand-clean, captioned, sized for Instagram, Facebook, and the profile.

OneWave-AI/claude-skills

Build a personalized product-demo or sales-walkthrough video in HyperFrames for a specific prospect or account — narrated screen-by-screen, branded to them, ending in a clear next step. Turns a generic demo into a "this was made for me" asset for outbound, follow-up, or deal acceleration. Use when you want to send a tailored video instead of booking another live call.

OneWave-AI/claude-skills

Turn real customer reviews, quotes, and case-study results into polished social-proof videos built in HyperFrames -- single-testimonial spotlights, review-wall montages, and stat-driven proof reels with captions and voiceover, cut for vertical, square, and wide. The highest-converting content most businesses never make.

OneWave-AI/claude-skills

Audit a codebase for LLM calls that are really classifications in disguise, then produce a costed swap plan for a System One model. Use when asked to cut AI inference cost or latency, when scoping a performance engagement for a client, when reviewing an agent loop that feels slow, or when asked "where could we use Jev here". Produces a ranked table of candidates with measured latency and dollar deltas.

OneWave-AI/claude-skills

Wire a System One model (Jev, or an open reproduction like Von) into a product feature — routing, guardrails, scoring, classification. Use when replacing an LLM call that returns a label rather than prose, when adding a typed decision to an agent loop, or when deciding between the hosted Jev API and a local open model. Covers question design, the eval-set-first workflow, threshold calibration, confidence gates, and the traps measured on real data.

OneWave-AI/claude-skills

Optimize landing pages for conversions, performance, and SEO. Use when improving landing pages, increasing conversions, or optimizing page performance.

OneWave-AI/claude-skills

TAM/SAM/SOM calculator with deep market research. Produces comprehensive market-sizing.md with top-down and bottom-up estimates, methodology, data sources, assumptions, sensitivity ranges, growth projections, competitive landscape, and Mermaid visualizations. Use when user needs market size estimates, addressable market analysis, go-to-market sizing, investor-ready market analysis, or business plan market validation.

OneWave-AI/claude-skills

Generate unique personalized first lines for hundreds of prospects using company news, LinkedIn activity, and mutual connections. Saves 10+ hours of manual research per campaign. Use when you need personalized outreach at volume.

OneWave-AI/claude-skills

Review investment portfolios for risk, diversification, fees. Asset allocation recommendations, tax-loss harvesting, rebalancing.

OneWave-AI/claude-skills

Create multiple choice, true/false, fill-in-blank, matching quizzes. Auto-generate plausible distractors. Instant grading with explanations.

OneWave-AI/claude-skills

Enhanced skill navigator that maps conversation history, recommends multi-skill chains, identifies patterns from past usage, and learns from session outcomes. Goes beyond basic scout with deep context analysis and workflow orchestration.

相關技能