CommunityCodierung & Entwicklunggithub.com

grahama1970/benchmark-models

Standardisierte Compliance-QRA-Benchmarks gegen Kandidaten-LLMs zur Bewertung von Genauigkeit, Latenz und Kosten.

Was ist benchmark-models?

Dieses Skill führt standardisierte Compliance-QRA-Tests gegen verschiedene LLMs durch, um deren Genauigkeit, Latenz und geschätzte Token-Kosten zu messen. Es ist besonders nützlich für Entwickler und KI-Ingenieure, die vor dem Einsatz eines Modells in der Inferenz-Pipeline eine fundierte Modellauswahl treffen müssen. Das Skill unterstützt Einzelmodell-Benchmarks sowie den direkten Vergleich mehrerer Modelle und gibt übersichtliche Ergebnis-Tabellen aus. Es arbeitet über das Skript `run.sh` und speichert die Ergebnisse in `~/.embry/benchmark_results.json`. Der Trockenlauf ermöglicht Tests ohne echte LLM-Aufrufe. Die standardisierten Test-Suites decken wichtige Compliance-Rahmenwerke wie NIST 800-171, AS9100D, CMMC, ITAR, DFARS, DO-178C, MIL-STD sowie die Erkennung von Compliance-Abweichungen zwischen Programmen ab.

Funktioniert mit~Claude Code~Codex CLI~Cursor✓Antigravity
Part ofAgent Workflows
npx skills add https://github.com/grahama1970/agent-skills/tree/main/skills/benchmark-models

Installed? Explore more Codierung & Entwicklung skills: steipete/bluebubbles, steipete/eightctl, steipete/blucli · View all 6 →

Original anzeigen→Alle Skills durchsuchen

In Ihrer bevorzugten KI fragen

Öffnet einen neuen Chat, in dem dieser Agent-Skill bereits geladen ist.

OpenAIChatGPTClaudeClaudeGeminiGeminiGrokGrokPerplexityPerplexityDeepSeekDeepSeek

Dokumentation

Was macht benchmark-models?

Führen Sie standardisierte Compliance-QRA-Tests gegen Kandidaten-LLMs durch, um Genauigkeit, Latenz und Kosten vor dem Einsatz in der Inferenz-Pipeline zu bewerten.

Verwendung

Einzelmodell-Benchmark ausführen

./run.sh run --model deepseek-v3 --suite compliance-basic

Führt 20 Goldstandard-Compliance-QRA-Fragen gegen das angegebene Modell über /scillm aus. Gibt eine Ergebnistabelle mit den Spalten TEST_CASE, EXPECTED, ACTUAL, MATCH und LATENCY_MS sowie zusammenfassende Metriken (Genauigkeit%, Latenz_p50, Latenz_p95, geschätzte Token-Kosten) aus.

Mehrere Modelle vergleichen

./run.sh compare --models "deepseek-v3,llama-3.1-70b" --suite compliance-basic

Führt den Benchmark für jedes Modell aus und gibt eine nebeneinander angeordnete Vergleichstabelle aus.

Letzten Bericht anzeigen

./run.sh report

Liest die aktuellsten Benchmark-Ergebnisse aus ~/.embry/benchmark_results.json und gibt eine Zusammenfassung aus.

Trockenlauf (keine LLM-Aufrufe)

./run.sh run --model deepseek-v3 --suite compliance-basic --dry-run
./run.sh compare --models "deepseek-v3,llama-3.1-70b" --dry-run

Gibt das vollständige Benchmark-Gerüst mit 20 Testfällen und simulierten Ergebnissen aus. Es werden keine LLM-Aufrufe getätigt.

Suites

  • compliance-basic: 20 Goldstandard-QRA-Fragen zu NIST 800-171, AS9100D, CMMC, ITAR, DFARS, DO-178C, MIL-STD und programmübergreifender Compliance-Abweichungserkennung.

Ausgabe

Die Ergebnisse werden in ~/.embry/benchmark_results.json gespeichert und auf der Standardausgabe ausgegeben. Der Bericht enthält die Genauigkeit pro Frage sowie zusammenfassende Metriken für Modellauswahlentscheidungen.

Individual skills in this repo

This repo contains 20 individual skills — each has its own dedicated page.

grahama1970/acceptance-contract

Turn a client brief, zip bundle, directory, or single requirements file into a typed acceptance-contract bundle with extracted requirements, acceptance checks, open questions, an immutable-goal draft, and a create-report-backed decision report. Use when users say acceptance contract, brief to requirements, freeze the goal, create immutable goal, amend immutable goal, build a Battle requirements bundle, or extract requirements from this bundle.

grahama1970/agent-ecosystem

Canonical map and shared contracts for the agent-governance ecosystem: the pi.receipt_envelope.v1 boundary envelope, the component graph, and the rules for which component owns which schema. Use when wiring a skill or extension into the shared receipt world, when asking how shame, triage-error, tau, ask, project-watchdog, ops-herdr, ponytail, and Memory fit together, or when validating an envelope.

grahama1970/agentic-evals

Agentic evaluation of skills using multi-trial fixtures, deterministic command assertions, trajectory checks, safety constraints, and evidence-backed readiness scoring. Use when users ask for agentic evals, multi-trial skill evaluation, skill trajectory validation, or readiness scoring for a skill workflow.

grahama1970/agent-inbox

File-based inter-agent messaging with headless dispatch. Check inbox, send bugs/requests to other projects, automatically spawn headless agents to fix bugs, and track progress via task-monitor.

grahama1970/agents-registry

Generate and query the centralized agent identity registry. Scans .pi/agents/*/AGENTS.md, parses frontmatter, outputs agents-registry.json and optionally syncs to /memory for semantic search.

grahama1970/agent-status

Artifact-driven status surfaces for long-running project-agent work. Maintains status.json, events.jsonl, proof manifests, and a stale-aware STATUS.html so humans can tell where the agent is, what passed, what is still unproven, and what decision or action is next — without dashboard theater.

grahama1970/align

Round-based context alignment before execution. Use when the human, project agent, WebGPT, scillm, ask, dogpile, memory, or project-knowledge may each hold different facts about a task; especially before ambiguous design, infographic, product workflow, high-stakes implementation, plan-iterate, project-infographic, or multi-review work.

grahama1970/analytics

Flexible data science analytics for any dataset. Auto-discovers schema, recommends charts, exports to create-figure. Works with JSONL, JSON, CSV from any source.

grahama1970/analyze-chatterbox-emotions

Evaluate generated Chatterbox voice files as voice-quality artifacts: affect match, arousal/valence proxies, pause placement, intelligibility inputs, clipping, loudness, and discontinuity flags. Use when reviewing Chatterbox emotional tags, pauses, Turbo/base affect delivery, Persona Dream utterance renders, or whether generated speech matches an intended product-facing affect.

grahama1970/analyze-elf

Reverse-engineer features from ELF binaries. Extracts CLI commands, state machines, protocols, Zod schemas, and data models. Automatically generates a /create-walkthrough prosecution brief with Mermaid diagrams. Uses /treesitter for AST analysis of bundled JS/TS source.

grahama1970/animation-vocabulary

Reverse-lookup glossary that turns a vague description of a web animation or motion effect into its exact term ("the bouncy thing when a popover opens" → Pop in; "the iOS rubber-band scroll" → Rubber-banding). Use when the user asks "what's it called when…", or describes a motion effect without knowing its name and wants the right word to prompt an AI or designer with. For naming an effect, not designing or building one.

grahama1970/anonymize-data

Anonymize supported CSV, JSON, UTF-8 text, and SQLite files using an explicit policy through the oai-trial project. Use for anonymize data, pseudonymize exports, redact policy literals, or discover and explicitly approve fuzzy name aliases. The skill is a thin CLI/Docker interface, not another engine.

grahama1970/anvil

Heavy-duty "No-Vibes" debugging and hardening orchestrator. Use this for complex, stubborn bugs where `review-code` has failed, or for "Red Teaming" (hardening) a codebase. Runs multiple agents in parallel (Thunderdome) using git worktree isolation.

grahama1970/apple-design

Apple's approach to interface design and fluid, physical motion, translated for the web. Use when building or reviewing gesture-driven UI, spring animations, drag/swipe/sheet interactions, momentum and interruptible transitions, translucent materials and depth, typography (optical sizing, tracking, leading), reduced-motion, or the design foundations (feedback, spatial consistency, restraint) behind Apple-style interfaces.

grahama1970/argue

Multi-persona structured debate orchestrator. Personas research via /dogpile, consult colleagues via /ask, and argue toward nuanced synthesis on complex questions.

grahama1970/arxiv

Search arXiv for papers and extract knowledge into memory. Use `search` to find papers, `learn` to extract knowledge.

grahama1970/ask

Use when the user asks to query project memory, ask an oracle, use supported browser-backed reviewers, run Tau roundtable/single-handler workflows, ask Pi-native subagents from within Pi, run persona/deep-review workflows, generate image prompts, check OS/project health through composed skills, or run an ask DAG. This skill is the executable /ask runtime; do not replace it with an informal subagent, plain web search, or hand-written review; inside Pi, explicit Pi-native subagent targets are routed through the pi-subagents tool as an Ask target type.

grahama1970/assess

Step back and critically reassess project state. Use when asked to "assess", "step back", "fresh eyes", "check alignment", "sanity check", "health check", "prune documentation", or "evaluate what's working". Offers documentation pruning and doc-code alignment analysis. Offer to run after major changes (don't auto-run).

grahama1970/assistant

Shared GPT + classifier inference gateway for persona monitor tasks. Routes validation and classification through a 4-tier cascade: heuristic → classifier → local GPT → scillm.

grahama1970/assistant-lab

Self-improvement workbench for /assistant. All the tools needed to diagnose, train, evaluate, and promote models in a continuous loop. The "warm pond" where /assistant evolves its own inference stack.

Verwandte Skills

steipete/bluebubbles

Send and manage iMessages via BlueBubbles, including attachments, tapbacks, edits, replies, and groups.

community

steipete/eightctl

Control Eight Sleep pods (status, temperature, alarms, schedules).

community

steipete/blucli

BluOS CLI (blu) for discovery, playback, grouping, and volume.

community

steipete/bear-notes

Create, search, and manage Bear notes via grizzly CLI.

community

steipete/camsnap

Capture frames or clips from RTSP/ONVIF cameras.

community

steipete/gifgrep

Search GIF providers with CLI/TUI, download results, and extract stills/sheets.

community
← More Codierung & Entwicklung skills