Community编程与开发github.com

grahama1970/基准测试模型

针对候选LLM运行标准化的合规QRA基准测试,以评估准确性、延迟和成本。

基准测试模型 是什么?

这是一个AI代理技能,用于在部署前对候选大语言模型(LLM)进行标准化合规快速问答(QRA)基准测试。它帮助开发者通过一致性测试集评估模型质量、延迟和成本,从而做出更明智的模型选择决策。该技能支持多种模型进行单次测试或横向对比,并生成包含准确性、延迟百分位和估计令牌成本的综合报告。它通过`/scillm` API或模拟结果运行,适用于Claude Code、Cursor、Codex等平台。主要功能包括运行基准测试、对比模型、查看报告以及无LLM调用的干运行。

兼容平台~Claude Code~Codex CLI~Cursor✓Antigravity
npx skills add https://github.com/grahama1970/agent-skills/tree/main/skills/benchmark-models

Installed? Explore more 编程与开发 skills: steipete/bluebubbles, steipete/eightctl, steipete/blucli · View all 6 →

在你喜欢的 AI 中提问

打开一个已预加载此 Agent Skill 的新对话。

文档

benchmark-models

运行标准化合规QRA测试,针对候选LLM评估准确性、延迟和成本,然后再部署到推理管道中。

用法

运行单个模型基准测试

./run.sh run --model deepseek-v3 --suite compliance-basic

针对指定模型(通过/scillm)运行20道金标准合规QRA问题。输出包含TEST_CASE、EXPECTED、ACTUAL、MATCH和LATENCY_MS列的结果表,以及汇总指标(准确率%、延迟_p50、延迟_p95、估计令牌成本)。

对比多个模型

./run.sh compare --models "deepseek-v3,llama-3.1-70b" --suite compliance-basic

对每个模型运行基准测试,并输出并排对比表。

查看最新报告

./run.sh report

从~/.embry/benchmark_results.json读取最近的基准测试结果,并渲染摘要。

干运行(无LLM调用)

./run.sh run --model deepseek-v3 --suite compliance-basic --dry-run
./run.sh compare --models "deepseek-v3,llama-3.1-70b" --dry-run

输出完整的基准测试框架,包含20个测试案例和模拟结果。不进行任何LLM调用。

测试套件

  • compliance-basic: 20道金标准QRA问题,涵盖NIST 800-171、AS9100D、CMMC、ITAR、DFARS、DO-178C、MIL-STD以及跨项目合规漂移检测。

输出

结果保存到~/.embry/benchmark_results.json,并打印到标准输出。报告包含每个问题的准确率以及用于模型选择决策的汇总指标。

Individual skills in this repo

This repo contains 20 individual skills — each has its own dedicated page.

grahama1970/acceptance-contract

Turn a client brief, zip bundle, directory, or single requirements file into a typed acceptance-contract bundle with extracted requirements, acceptance checks, open questions, an immutable-goal draft, and a create-report-backed decision report. Use when users say acceptance contract, brief to requirements, freeze the goal, create immutable goal, amend immutable goal, build a Battle requirements bundle, or extract requirements from this bundle.

grahama1970/agent-ecosystem

Canonical map and shared contracts for the agent-governance ecosystem: the pi.receipt_envelope.v1 boundary envelope, the component graph, and the rules for which component owns which schema. Use when wiring a skill or extension into the shared receipt world, when asking how shame, triage-error, tau, ask, project-watchdog, ops-herdr, ponytail, and Memory fit together, or when validating an envelope.

grahama1970/agentic-evals

Agentic evaluation of skills using multi-trial fixtures, deterministic command assertions, trajectory checks, safety constraints, and evidence-backed readiness scoring. Use when users ask for agentic evals, multi-trial skill evaluation, skill trajectory validation, or readiness scoring for a skill workflow.

grahama1970/agent-inbox

File-based inter-agent messaging with headless dispatch. Check inbox, send bugs/requests to other projects, automatically spawn headless agents to fix bugs, and track progress via task-monitor.

grahama1970/agents-registry

Generate and query the centralized agent identity registry. Scans .pi/agents/*/AGENTS.md, parses frontmatter, outputs agents-registry.json and optionally syncs to /memory for semantic search.

grahama1970/agent-status

Artifact-driven status surfaces for long-running project-agent work. Maintains status.json, events.jsonl, proof manifests, and a stale-aware STATUS.html so humans can tell where the agent is, what passed, what is still unproven, and what decision or action is next — without dashboard theater.

grahama1970/align

Round-based context alignment before execution. Use when the human, project agent, WebGPT, scillm, ask, dogpile, memory, or project-knowledge may each hold different facts about a task; especially before ambiguous design, infographic, product workflow, high-stakes implementation, plan-iterate, project-infographic, or multi-review work.

grahama1970/analytics

Flexible data science analytics for any dataset. Auto-discovers schema, recommends charts, exports to create-figure. Works with JSONL, JSON, CSV from any source.

grahama1970/analyze-chatterbox-emotions

Evaluate generated Chatterbox voice files as voice-quality artifacts: affect match, arousal/valence proxies, pause placement, intelligibility inputs, clipping, loudness, and discontinuity flags. Use when reviewing Chatterbox emotional tags, pauses, Turbo/base affect delivery, Persona Dream utterance renders, or whether generated speech matches an intended product-facing affect.

grahama1970/analyze-elf

Reverse-engineer features from ELF binaries. Extracts CLI commands, state machines, protocols, Zod schemas, and data models. Automatically generates a /create-walkthrough prosecution brief with Mermaid diagrams. Uses /treesitter for AST analysis of bundled JS/TS source.

grahama1970/animation-vocabulary

Reverse-lookup glossary that turns a vague description of a web animation or motion effect into its exact term ("the bouncy thing when a popover opens" → Pop in; "the iOS rubber-band scroll" → Rubber-banding). Use when the user asks "what's it called when…", or describes a motion effect without knowing its name and wants the right word to prompt an AI or designer with. For naming an effect, not designing or building one.

grahama1970/anonymize-data

Anonymize supported CSV, JSON, UTF-8 text, and SQLite files using an explicit policy through the oai-trial project. Use for anonymize data, pseudonymize exports, redact policy literals, or discover and explicitly approve fuzzy name aliases. The skill is a thin CLI/Docker interface, not another engine.

grahama1970/anvil

Heavy-duty "No-Vibes" debugging and hardening orchestrator. Use this for complex, stubborn bugs where `review-code` has failed, or for "Red Teaming" (hardening) a codebase. Runs multiple agents in parallel (Thunderdome) using git worktree isolation.

grahama1970/apple-design

Apple's approach to interface design and fluid, physical motion, translated for the web. Use when building or reviewing gesture-driven UI, spring animations, drag/swipe/sheet interactions, momentum and interruptible transitions, translucent materials and depth, typography (optical sizing, tracking, leading), reduced-motion, or the design foundations (feedback, spatial consistency, restraint) behind Apple-style interfaces.

grahama1970/argue

Multi-persona structured debate orchestrator. Personas research via /dogpile, consult colleagues via /ask, and argue toward nuanced synthesis on complex questions.

grahama1970/arxiv

Search arXiv for papers and extract knowledge into memory. Use `search` to find papers, `learn` to extract knowledge.

grahama1970/ask

Use when the user asks to query project memory, ask an oracle, use supported browser-backed reviewers, run Tau roundtable/single-handler workflows, ask Pi-native subagents from within Pi, run persona/deep-review workflows, generate image prompts, check OS/project health through composed skills, or run an ask DAG. This skill is the executable /ask runtime; do not replace it with an informal subagent, plain web search, or hand-written review; inside Pi, explicit Pi-native subagent targets are routed through the pi-subagents tool as an Ask target type.

grahama1970/assess

Step back and critically reassess project state. Use when asked to "assess", "step back", "fresh eyes", "check alignment", "sanity check", "health check", "prune documentation", or "evaluate what's working". Offers documentation pruning and doc-code alignment analysis. Offer to run after major changes (don't auto-run).

grahama1970/assistant

Shared GPT + classifier inference gateway for persona monitor tasks. Routes validation and classification through a 4-tier cascade: heuristic → classifier → local GPT → scillm.

grahama1970/assistant-lab

Self-improvement workbench for /assistant. All the tools needed to diagnose, train, evaluate, and promote models in a continuous loop. The "warm pond" where /assistant evolves its own inference stack.

相关技能