Communitygithub.com

OneWave-AI/claude-skills

为System One模型(Jev、Von或任何类型决策配置)构建并运行带标签的评估集,然后对标准措辞和阈值进行扫描。当Jev/Von分类错误或不可靠、在托管API与本地开源模型之间选择、调整noul阈值或即将发布任何类型决策功能时使用。生成准确性-措辞矩阵和校准阈值。

claude-skills 是什么?

该技能用于评估System One类型决策模型的准确性,特别是Jev、Von或自定义配置。其核心思想是,评估集本身才是产品——模型输出的失败模式是静默错误,没有标签就无法区分是问题措辞不当还是模型本身的缺陷。通过构建带标签的评估集(至少50条记录,发布前达200条以上),然后扫描多种不同的标准措辞和阈值,技能可以生成一个准确性-措辞矩阵,清晰展示不同模型和措辞下的表现。该技能支持Jev、Von、Laya等多种后端,并提供阈值扫描和置信度门控评估。实际使用表明,措辞变化可导致结果从4/15提升至14/15,但小样本集可能产生误导性结论。技能会输出每个配置的准确率、跨配置的稳定性(ROBUST或FRAGILE)、校准后的阈值以及门控效果。

兼容平台✓Claude Code~Codex CLI~Cursor
npx skills add https://github.com/OneWave-AI/claude-skills/tree/HEAD/jev-eval

在你喜欢的 AI 中提问

打开一个已预加载此 Agent Skill 的新对话。

文档

评估类型决策配置

评估集即产品。 System One模型的准确性在很大程度上取决于问题的编写方式,其失败模式是静默的——它返回一个自信、类型有效但错误的答案。没有标签,你无法区分是问题糟糕还是模型糟糕。

实测结果:重写标准措辞后,一个开源模型在15条记录上的得分从4/15提升至14/15。模型未变。随后在150条记录上做相同对比,该模型整体准确率为61%,而Jev为97%——15条记录的读数只是小样本且易于集所带来的假象。这两个事实都说明问题:措辞会显著影响结果,而小样本集会误导方向。

运行

python ~/.claude/skills/jev-eval/scripts/sweep.py labelled.json configs.json \
    --backend jev|von --question <name>

labelled.json 格式为 [{"id","state","truth"}]。configs.json 将配置名称映射到 {"instructions", "criteria"}——字典格式的选项表示选择,列表形式的级别表示评分,省略则表示noul。脚本从钥匙串(typesafe-api-key)中读取Jev密钥,打印每个配置的准确率,将差异标记为ROBUST或FRAGILE,扫描noul的阈值,并对置信度门控进行评分。

实际输出(50条代理shell命令风险记录,仅后端不同):

config                         准确率   毫秒/条
A 简洁单行描述                   45/50       410      <- Jev
B 丰富标准                      49/50       415
C 丰富标准 + 排除项              46/50       418
D 故意简单描述                    44/50       411
差异范围:44/50 至 49/50   (ROBUST - 措辞不是关键)

A 简洁单行描述                     9/50        66      <- Von,相同配置
B 丰富标准                        23/50       121
C 丰富标准 + 排除项                15/50       129
D 故意简单描述                    22/50        53
差异范围:9/50 至 23/50    (FRAGILE - 但天花板仍然不可用)

先看天花板再看差异范围。一个FRAGILE模型,其最佳配置仅为23/50,这不是你通过修改措辞能解决的问题——这是在那个问题上选错了模型。

1. 构建评估集

最少50条记录,发布前至少200条。 从真实数据流中提取,而非合成数据。

  • 包含无聊的中等样本,而不仅仅是清晰的示例和戏剧性的边缘案例
  • 包含元数据损坏/缺失的记录——这正是分类器会失败的地方
  • 通过阅读记录来标注,在任何模型运行之前。切勿根据模型输出进行标注
  • 以JSON格式存储,包含原始记录和一个 truth 字段
[{"id":"D-1994","state":"...完整记录文本...","truth":"inbound_prospect"}]

如果你自己都无法自信地标注一条记录,模型也不能——要么删除它,要么修改问题以使答案具有确定性。

2. 扫描措辞,而非仅模型

核心操作。编写3–4种真正不同的标准配置,并全部运行:

  • A — 每个选项一行简洁描述(大多数人首先会写的)
  • B — 每个选项3–4句话,附带具体示例
  • C — B的基础上加上明确的默认值和明确的排除项("仅当……时")
  • D — 故意简单描述,四五个词,作为底线测试

报告每个模型下各配置的准确率:

配置                      JEV       VON      LAYA     (主分类,n=50)
A 简洁单行描述           47/50     34/50     21/50
B 丰富标准              49/50     28/50     15/50
C 丰富标准 + 排除项      48/50     24/50     19/50
D 故意简单描述          48/50     22/50     24/50

先看底线,再看差异范围。 底线是“如果我表达得很糟糕,这个模型还能不能完成任务”;差异范围是“维护它需要付出多大代价。”根据在命令任务上的实测,每个托管模型的底线在82-92%(Haiku 46-48,GPT-4.1-mini 45-50,Jev 44-49,GPT-5-mini 41-49),而Von底线为9/50,Laya为18/50。注意,Jev在措辞鲁棒性上并不比小型LLM更强——它同样会波动约十个百分点。你购买的是底线,而非免疫力。

注意主分类列并未展示出一个清晰的“更丰富更好”的梯度。Von在此处的最佳配置是简洁版本。任何能提高开源模型数值的因素,都是对表面形式的敏感性,而非理解能力,所以不要认为你的下一次标准重写会提升它——重新运行评估集。

3. 为每个noul扫描阈值

永远不要直接发布0.5。扫描并读取曲线:

for t in [0.5,0.6,0.7,0.75,0.8,0.85,0.9,0.95]:
    tp = sum(p>=t and y     for p,y in z); fp = sum(p>=t and not y for p,y in z)
    fn = sum(p< t and y     for p,y in z); tn = sum(p< t and not y for p,y in z)
    print(f"{t}  P={tp/max(tp+fp,1):.2f}  R={tp/max(tp+fn,1):.2f}  acc={(tp+tn)/len(z):.2f}")

不同模型具有不同的底线。Jev的noul在明显干净的记录上落在0.2–0.5,而Claude则降至0.0——因此其实际截断点约为0.85。在一个模型上调优的阈值不能迁移到另一个模型上。切换模型时重新扫描。

4. 诚实评估置信度门控

两个数字,始终同时给出:

errs   = [r for r in rows if r.pred != r.truth]
rights = [r for r in rows if r.pred == r.truth]
for g in [0.5,0.7,0.9]:
    caught    = sum(r.conf <  g for r in errs)      # 门控捕获的错误数
    escalated = sum(r.conf <  g for r in rows)      # 升级的总量
    print(f"门控 {g}: 捕获 {caught}/{len(errs)} 个错误,升级 {escalated/len(rows):.0%} 的流量")

一个门控捕获了11个错误中的10个,但升级了93%的流量,这不是一个有效的门控——它只是一个多了一层步骤的慢速路径。良好的校准若不伴随良好的准确率毫无价值。

5. 报告

  • 每个模型下各配置的准确率,明确指出差异范围
  • 每个noul选择的阈值,附带证明该选择合理的扫描结果
  • 门控:捕获的错误数 和 升级的流量比例
  • 在生产规模下每1k条记录的预估延迟和成本
  • 明确说明评估集的大小及其未覆盖的内容

小样本集会撒谎。如果两条模型在15条记录上都得到100%准确率,那什么都无法区分——直接说明这一点,而非暗示该平局有意义。

相关

jev-integrate 是本评估流程所支撑的工作流。jev-audit 用于发现值得评估的候选对象。

Individual skills in this repo

This repo contains 20 individual skills — each has its own dedicated page.

OneWave-AI/claude-skills

Deploy a 2-layer parallel agent hierarchy for large, parallelizable work — big refactors, multi-file migrations, codebase-wide audits, bulk generation. A top-tier commander (Fable or Opus) orchestrates the swarms; the user picks a power level (Max Power / Heavy / Balanced / Economy) that sets the Opus/Sonnet/Haiku model mix per layer. Layer 1 is 3-50+ specialist agents, each with its own full context window; Layer 2 is 2+ sub-agents per member. Includes git safety, tiered sizing, a pre-deploy gate, phantom-completion checks, and multi-wave follow-up.

OneWave-AI/claude-skills

Generate animated videos and motion graphics from natural language descriptions. Creates a standalone Vite + React project with Framer Motion scenes that auto-play in the browser. Use when the user wants to create animations, motion graphics, video intros, animated presentations, or product demos.

OneWave-AI/claude-skills

Use before starting a product, feature, MVP, agent, automation, or integration build to identify the main pre-build risk and choose the smallest validation step. Trigger when the user asks whether to build, scope, validate, prioritize, or sanity-check an idea before implementation.

OneWave-AI/claude-skills

Post-mortem analysis when a client churns. Takes client history, engagement data, support tickets, usage logs, and exit feedback to produce a comprehensive churn autopsy with root cause classification, timeline of decline, and preventive measures.

OneWave-AI/claude-skills

Compose a premium, animated landing page section by section in Next.js + Tailwind + Motion (Framer Motion) — built on a real design system, with editorial copy and motion that feels human-crafted, not template-generated. Use when building or rebuilding a landing page, marketing site, or hero/feature/pricing/CTA sections that need to look production-ready and bespoke from day one.

OneWave-AI/claude-skills

Assemble 2-3 complementary experts to collaboratively analyze anything. Experts work together to explore topics from multiple expert angles.

OneWave-AI/claude-skills

Convert any topic into playable browser games. Types: trivia, matching, word puzzles, adventure games. Uses Phaser.js or Kaboom.js.

OneWave-AI/claude-skills

Turn a marketing brief or product offer into a finished short-form video ad built in HyperFrames — hook, script, shot-by-shot storyboard, scene-by-scene HTML composition, captions, and voiceover. Outputs platform cuts (vertical 9:16, square, 16:9). Use when you need a launch ad, product promo, social video, or paid-creative concept ready to render.

OneWave-AI/claude-skills

Turn a website URL, product docs, or a founder's rambling description into a tight 60-90 second explainer video built in HyperFrames -- problem/solution narrative, animated product walkthrough scenes, captions, and voiceover, ready for the homepage hero, sales emails, and onboarding.

OneWave-AI/claude-skills

Build promo videos for local businesses in HyperFrames -- the new-menu drop, the seasonal offer, the event announcement, the "we're hiring" post, the Google Business Profile clip. Restaurant, salon, gym, trades: 15-30 seconds, brand-clean, captioned, sized for Instagram, Facebook, and the profile.

OneWave-AI/claude-skills

Build a personalized product-demo or sales-walkthrough video in HyperFrames for a specific prospect or account — narrated screen-by-screen, branded to them, ending in a clear next step. Turns a generic demo into a "this was made for me" asset for outbound, follow-up, or deal acceleration. Use when you want to send a tailored video instead of booking another live call.

OneWave-AI/claude-skills

Turn real customer reviews, quotes, and case-study results into polished social-proof videos built in HyperFrames -- single-testimonial spotlights, review-wall montages, and stat-driven proof reels with captions and voiceover, cut for vertical, square, and wide. The highest-converting content most businesses never make.

OneWave-AI/claude-skills

Audit a codebase for LLM calls that are really classifications in disguise, then produce a costed swap plan for a System One model. Use when asked to cut AI inference cost or latency, when scoping a performance engagement for a client, when reviewing an agent loop that feels slow, or when asked "where could we use Jev here". Produces a ranked table of candidates with measured latency and dollar deltas.

OneWave-AI/claude-skills

Wire a System One model (Jev, or an open reproduction like Von) into a product feature — routing, guardrails, scoring, classification. Use when replacing an LLM call that returns a label rather than prose, when adding a typed decision to an agent loop, or when deciding between the hosted Jev API and a local open model. Covers question design, the eval-set-first workflow, threshold calibration, confidence gates, and the traps measured on real data.

OneWave-AI/claude-skills

Optimize landing pages for conversions, performance, and SEO. Use when improving landing pages, increasing conversions, or optimizing page performance.

OneWave-AI/claude-skills

TAM/SAM/SOM calculator with deep market research. Produces comprehensive market-sizing.md with top-down and bottom-up estimates, methodology, data sources, assumptions, sensitivity ranges, growth projections, competitive landscape, and Mermaid visualizations. Use when user needs market size estimates, addressable market analysis, go-to-market sizing, investor-ready market analysis, or business plan market validation.

OneWave-AI/claude-skills

Generate unique personalized first lines for hundreds of prospects using company news, LinkedIn activity, and mutual connections. Saves 10+ hours of manual research per campaign. Use when you need personalized outreach at volume.

OneWave-AI/claude-skills

Review investment portfolios for risk, diversification, fees. Asset allocation recommendations, tax-loss harvesting, rebalancing.

OneWave-AI/claude-skills

Create multiple choice, true/false, fill-in-blank, matching quizzes. Auto-generate plausible distractors. Instant grading with explanations.

OneWave-AI/claude-skills

Enhanced skill navigator that maps conversation history, recommends multi-skill chains, identifies patterns from past usage, and learns from session outcomes. Goes beyond basic scout with deep context analysis and workflow orchestration.

相关技能