Community程式設計與開發github.com

grahama1970/benchmark-models

針對候選LLM執行標準化合規QRA基準測試,評估準確度、延遲與成本。

benchmark-models 是什麼?

此技能提供一個標準化的基準測試框架,用於評估大型語言模型在合規性問答(QRA)任務上的表現。它支援單一模型執行與多模型比較,並輸出包含準確率、延遲百分位數(p50/p95)及預估 Token 成本的詳細報告,幫助開發者在部署前做出資料驅動的模型選擇決策。該工具設計為與 /scillm 模型推論端點整合,並可透過 --dry-run 選項進行無實際 LLM 呼叫的預演。支援的測試套件包含 compliance-basic,涵蓋 NIST 800-171、AS9100D、CMMC、ITAR、DFARS、DO-178C、MIL-STD 等法規標準的 20 道黃金題庫。結果會儲存在 ~/.embry/benchmark_results.json,並以表格形式輸出至終端機。

相容平台~Claude Code~Codex CLI~Cursor✓Antigravity
npx skills add https://github.com/grahama1970/agent-skills/tree/main/skills/benchmark-models

Installed? Explore more 程式設計與開發 skills: steipete/bluebubbles, steipete/eightctl, steipete/blucli · View all 6 →

在你喜歡的 AI 中提問

開啟一個已預先載入此 Agent Skill 的新對話。

說明文件

benchmark-models 是做什麼的?

在部署至推論管線前,針對候選LLM執行標準化合規QRA測試,以評估準確度、延遲與成本。

使用方式

執行單一模型基準測試

./run.sh run --model deepseek-v3 --suite compliance-basic

對指定模型(透過 /scillm)執行 20 道黃金題庫合規 QRA 問題。輸出結果表格包含 TEST_CASE、EXPECTED、ACTUAL、MATCH 與 LATENCY_MS 欄位,以及摘要指標(accuracy%、latency_p50、latency_p95、estimated_token_cost)。

比較多個模型

./run.sh compare --models "deepseek-v3,llama-3.1-70b" --suite compliance-basic

對每個模型執行基準測試,並輸出並排比較表格。

檢視上次報告

./run.sh report

讀取最近一次基準測試結果(來自 ~/.embry/benchmark_results.json)並呈現摘要。

乾執行(不呼叫LLM)

./run.sh run --model deepseek-v3 --suite compliance-basic --dry-run
./run.sh compare --models "deepseek-v3,llama-3.1-70b" --dry-run

輸出完整的基準測試框架(包含 20 個測試案例與 mock 結果),不實際呼叫任何 LLM。

測試套件

  • compliance-basic:20 道黃金題庫 QRA 問題,涵蓋 NIST 800-171、AS9100D、CMMC、ITAR、DFARS、DO-178C、MIL-STD 及跨計畫合規漂移檢測。

輸出

結果會儲存至 ~/.embry/benchmark_results.json 並輸出至終端機 stdout。報告包含每題的準確度與整體摘要指標,輔助模型選擇決策。

Individual skills in this repo

This repo contains 20 individual skills — each has its own dedicated page.

grahama1970/acceptance-contract

Turn a client brief, zip bundle, directory, or single requirements file into a typed acceptance-contract bundle with extracted requirements, acceptance checks, open questions, an immutable-goal draft, and a create-report-backed decision report. Use when users say acceptance contract, brief to requirements, freeze the goal, create immutable goal, amend immutable goal, build a Battle requirements bundle, or extract requirements from this bundle.

grahama1970/agent-ecosystem

Canonical map and shared contracts for the agent-governance ecosystem: the pi.receipt_envelope.v1 boundary envelope, the component graph, and the rules for which component owns which schema. Use when wiring a skill or extension into the shared receipt world, when asking how shame, triage-error, tau, ask, project-watchdog, ops-herdr, ponytail, and Memory fit together, or when validating an envelope.

grahama1970/agentic-evals

Agentic evaluation of skills using multi-trial fixtures, deterministic command assertions, trajectory checks, safety constraints, and evidence-backed readiness scoring. Use when users ask for agentic evals, multi-trial skill evaluation, skill trajectory validation, or readiness scoring for a skill workflow.

grahama1970/agent-inbox

File-based inter-agent messaging with headless dispatch. Check inbox, send bugs/requests to other projects, automatically spawn headless agents to fix bugs, and track progress via task-monitor.

grahama1970/agents-registry

Generate and query the centralized agent identity registry. Scans .pi/agents/*/AGENTS.md, parses frontmatter, outputs agents-registry.json and optionally syncs to /memory for semantic search.

grahama1970/agent-status

Artifact-driven status surfaces for long-running project-agent work. Maintains status.json, events.jsonl, proof manifests, and a stale-aware STATUS.html so humans can tell where the agent is, what passed, what is still unproven, and what decision or action is next — without dashboard theater.

grahama1970/align

Round-based context alignment before execution. Use when the human, project agent, WebGPT, scillm, ask, dogpile, memory, or project-knowledge may each hold different facts about a task; especially before ambiguous design, infographic, product workflow, high-stakes implementation, plan-iterate, project-infographic, or multi-review work.

grahama1970/analytics

Flexible data science analytics for any dataset. Auto-discovers schema, recommends charts, exports to create-figure. Works with JSONL, JSON, CSV from any source.

grahama1970/analyze-chatterbox-emotions

Evaluate generated Chatterbox voice files as voice-quality artifacts: affect match, arousal/valence proxies, pause placement, intelligibility inputs, clipping, loudness, and discontinuity flags. Use when reviewing Chatterbox emotional tags, pauses, Turbo/base affect delivery, Persona Dream utterance renders, or whether generated speech matches an intended product-facing affect.

grahama1970/analyze-elf

Reverse-engineer features from ELF binaries. Extracts CLI commands, state machines, protocols, Zod schemas, and data models. Automatically generates a /create-walkthrough prosecution brief with Mermaid diagrams. Uses /treesitter for AST analysis of bundled JS/TS source.

grahama1970/animation-vocabulary

Reverse-lookup glossary that turns a vague description of a web animation or motion effect into its exact term ("the bouncy thing when a popover opens" → Pop in; "the iOS rubber-band scroll" → Rubber-banding). Use when the user asks "what's it called when…", or describes a motion effect without knowing its name and wants the right word to prompt an AI or designer with. For naming an effect, not designing or building one.

grahama1970/anonymize-data

Anonymize supported CSV, JSON, UTF-8 text, and SQLite files using an explicit policy through the oai-trial project. Use for anonymize data, pseudonymize exports, redact policy literals, or discover and explicitly approve fuzzy name aliases. The skill is a thin CLI/Docker interface, not another engine.

grahama1970/anvil

Heavy-duty "No-Vibes" debugging and hardening orchestrator. Use this for complex, stubborn bugs where `review-code` has failed, or for "Red Teaming" (hardening) a codebase. Runs multiple agents in parallel (Thunderdome) using git worktree isolation.

grahama1970/apple-design

Apple's approach to interface design and fluid, physical motion, translated for the web. Use when building or reviewing gesture-driven UI, spring animations, drag/swipe/sheet interactions, momentum and interruptible transitions, translucent materials and depth, typography (optical sizing, tracking, leading), reduced-motion, or the design foundations (feedback, spatial consistency, restraint) behind Apple-style interfaces.

grahama1970/argue

Multi-persona structured debate orchestrator. Personas research via /dogpile, consult colleagues via /ask, and argue toward nuanced synthesis on complex questions.

grahama1970/arxiv

Search arXiv for papers and extract knowledge into memory. Use `search` to find papers, `learn` to extract knowledge.

grahama1970/ask

Use when the user asks to query project memory, ask an oracle, use supported browser-backed reviewers, run Tau roundtable/single-handler workflows, ask Pi-native subagents from within Pi, run persona/deep-review workflows, generate image prompts, check OS/project health through composed skills, or run an ask DAG. This skill is the executable /ask runtime; do not replace it with an informal subagent, plain web search, or hand-written review; inside Pi, explicit Pi-native subagent targets are routed through the pi-subagents tool as an Ask target type.

grahama1970/assess

Step back and critically reassess project state. Use when asked to "assess", "step back", "fresh eyes", "check alignment", "sanity check", "health check", "prune documentation", or "evaluate what's working". Offers documentation pruning and doc-code alignment analysis. Offer to run after major changes (don't auto-run).

grahama1970/assistant

Shared GPT + classifier inference gateway for persona monitor tasks. Routes validation and classification through a 4-tier cascade: heuristic → classifier → local GPT → scillm.

grahama1970/assistant-lab

Self-improvement workbench for /assistant. All the tools needed to diagnose, train, evaluate, and promote models in a continuous loop. The "warm pond" where /assistant evolves its own inference stack.

相關技能