SkillFresh – Agent Capability Degradation & Fitness Evaluator

Skill Auditor — Agent Capability Depreciation & Fitness Evaluation

Overview

A skill is a depreciating asset. Model versions ship every few months. A skill that passed last quarter can silently regress — wrong output shape, broken tool calls, missed triggers — and nobody tells the user their agent got dumber.

This skill is the evaluation methodology. It answers three questions:

1. Is a skill degrading? — capability regression across model versions.
2. Is a new skill actually good? — effectiveness vs cost.
3. Does a skill fit THIS user? — trigger accuracy + environment + style.

It ships five executable entry points:

• scripts/console.py — the resident console (data layer): maintains registry.<host>.json per host, detects model-version changes (bump/check), and emits the recommend/eliminate panel. See USAGE.md for the full command set.
• scripts/regress.py — given golden-set replay results, compute the pass-rate matrix, flag degradation (threshold + direction), and flag FINGERPRINT-DRIFT when an asset was edited instead.
• scripts/baseline.py — scaffold a replay checklist, then merge recorded verdicts into baseline-results.json (by asset; --replace for a full overwrite). Draft golden cases are skipped unless --allow-draft.
• scripts/fingerprint.py — standalone report CLI; a thin wrapper over the same hashing code scan uses. It writes fingerprint-report.* only; it does not write the registry.
• scripts/install.py — install this skill into every agent host on the machine (~/.workbuddy-ai/skills, ~/.claude/skills, ~/.agents/skills, ~/.dsh/skills, ~/.codex/skills). Verifies after installing; never leaves an empty directory behind; never copies personal data.

Core principle: No measurement without a baseline. Degradation = significant pass-rate drop under an unchanged fingerprint — never "feels dumber".

Multi-Host Architecture (skills vs plugins)

One core engine, per-host adapters. The same fingerprint / baseline / regression / recommend / need-suggestion logic runs everywhere; an adapter only knows where assets live and how to read their manifest.

host	kind	default root	manifest
workbuddy	skill	skills root that contains this skill	SKILL.md
claude-code	skill	~/.claude/skills	SKILL.md
deepseek	plugin	~/.dsh/profiles/desktop (reads package.json deps)	package.json / plugin.json
generic	asset	pass --root	plugin.json / manifest.json / SKILL.md / package.json

• Each host keeps its own registry: registry.<host>.json — no cross-host overwrite.
• Baseline results + golden sets are shared, keyed by asset name.
• Run with --host <name> --root <path>; add custom hosts in hosts.json.
• console.py hosts shows every profile, its resolved root, and detected asset count.

So in WorkBuddy you evaluate skills; in DeepSeek harness you evaluate plugins — same commands, different adapter. Adding a new host is a hosts.json entry (kind + root + manifest filenames), no code change.

When to Use

• Before deploying a new skill or plugin ("Create a skill and follow the evaluation workflow").
• After a model-version bump, to check whether existing skills regressed.
• When a skill "feels off" — confirm with data, not vibes.
• When deciding whether to keep / rewrite / retire a skill.
• When recommending a skill or plugin to a user (fit + evidence).

When NOT to use: one-off tasks with no reusable skill; pure narrative docs.

Core Concept: The Version Fingerprint

The fingerprint is the row of the depreciation matrix.

fingerprint = hash(capability surface file content + relative path)        # spec v2-ability-surface

What counts as the capability surface (the files whose change actually changes behaviour):

• included: SKILL.md, scripts/*, lib/*, dist/*, package.json, plugin.json, cordis.patch.yml, templates
• excluded: README*, CHANGELOG*, LICENSE*, images/audio/fonts, *.map, lockfiles, node_modules/, test*/, docs/

Why exclude: a README edit does not change capability, but hashing it makes the fingerprint move and produces a fake FINGERPRINT-DRIFT ("the asset was edited") when nothing functional changed. Plugins make this worse — one installed plugin can carry 16 MB of bundled output plus a 57 KB README.

Rules:

• Fingerprint unchanged ⇒ the row persists; a pass-rate drop is model regression.
• Fingerprint changed ⇒ the change is the skill's own responsibility, not the model's. Old and new rows coexist, so skill edits are never mis-attributed to model drift.

Run python scripts/console.py scan to compute fingerprints and store them in the registry. fingerprint.py is a standalone report wrapper; it does not touch the registry.

Capability Surface (why plugins need a second axis)

The fingerprint says "did the files change". It cannot say "does it still offer the same thing". scan therefore also records a capability surface per asset and diffs it against the previous scan:

Asset kind	What the surface records
plugin	patch (dsh.bundle.patch), main/module, bin, exports, deps, skills, mcp, hooks, insert (cordis rows), scripts
skill	tools (allowed-tools count), scripts

When it changes, scan prints it explicitly:

[caps] 1 asset's capability surface changed (this means "what it exposes externally has changed", unrelated to pass rate):
       plug-a                                   +bin

Why this matters for plugins: a plugin degrades by not registering its tools / 404-ing its routes / not running its hooks — not by "answering worse". A golden set of prompts measures the wrong thing for a plugin. The surface diff catches "v1.1.0 dropped an entry point" without needing to boot the host.

Known boundary (do not paper over it): the surface is declared capability, not observed capability. For a plugin whose manifest lies (declares a patch that fails to load), this cannot catch it.

Runtime Evidence (the third axis, and what it cannot do)

There are two very different things called "a runtime probe". Do not conflate them:

	Active probe	Passive evidence — implemented
what it does	boot the host, ask "which tools did you actually register"	read the log the host already writes
cost	DSH desktop profile is Electron-exclusive — dsh --profile desktop --dump-config is refused; no headless entry → reverse-engineer IPC or drive the UI	one file read, ~10 ms
catches	everything, including a plugin whose manifest lies	anything the plugin logged
status	not built	console.py probe / skill_audit_probe

probe reads ~/.dsh/logs/*.log (configurable per host via "logs" in hosts.json), parses <iso> [level] [source] message, attributes lines whose source equals an installed asset name, and flags lines matching failure vocabulary (fail|error|ENOENT|unavailable|timeout|...).

Real catch, on this machine:

@wxg-prc-cpg/browser-skill-dsh-plugin   2 failure indicators
  bsk probe failed (spawn bsk ENOENT); browser tools will report install guidance…

Files present, manifest valid, caps=4 — every static check passes. The plugin is loaded and its tools are registered, but they cannot work: the bsk CLI it shells out to is not on PATH. No amount of file hashing finds this.

Blind spots (state them, never hide them): a plugin that logs nothing is invisible; log rotation erases old faults; a plugin nobody has used has no evidence; info-level faults are caught by keyword matching, so both misses and false positives are possible. Absence of evidence is not evidence of health — probe prints that sentence itself rather than a green check.

Implemented vs Methodology-Only (do not blur this line)

The methodology below is broader than the code. Users should know which half they are holding:

Area	Status
Fingerprint over the capability surface, scan/status, capability diff	implemented
Golden scaffold + baseline ingest (merge by asset, draft guard)	implemented (baseline.py)
Pass-rate matrix, ≥20pp threshold + direction, FINGERPRINT-DRIFT	implemented (regress.py)
Model-change detection (bump/check), per-host stamps	implemented — but the event source is manual; a scheduled check stays green until someone bumps
Recommend/eliminate panel, need suggestion, history-mined drafts	implemented (heuristic; suggestions are not evidence)
Runtime evidence from host logs	implemented (passive; only what the plugin logged)
Ablation, shadow-mode adoption rate, net-value / token telemetry	methodology only — no code
Trigger precision/recall, environment reconciliation, style baseline	methodology only — no code
K≥3 replays, LLM-judge cross-validation	not implemented; current verdicts are K=1 self-assessment
risk column	maintenance-surface proxy — not value

If a claim in this file cannot be traced to a row marked "implemented", treat it as advice for the human running the audit, not as something the tool will do for them.

Judgment 1 — Capability Degradation

Four steps:

1. Build the golden set from history, not by hand. Mine past sessions: user adopted without pushback → positive replay; user pushed back / asked rewrite → negative. Replay IS regression. The user's own usage history is the corpus — no invented test set.
2. Run both models on the same golden set, K≥3 (LLM-judge scoring cases: K≥5, because the judge is noisy). Record pass rate per (fingerprint, model version).
3. Decision rule: drop ≥20pp AND directionally consistent → report degradation; below threshold → "volatility watch". This turns "this output is weird" into a reproducible criterion.
4. Attribute, don't just flag. Inspect intermediate artifacts:
   • Tool call error / format changed → fix tool description.
   • Output structure right, content wrong → rewrite instructions.
   • Should-have-triggered but didn't → fix trigger description.
   • All intermediates right, only final wrong → model-internal regression; mark skill "single-model-dependent", wait for model fix.

The "7 degraded, 2 rewrite-recommended" line is the attributable subset of step 4's output — that is what makes the report actionable.

Judgment 2 — Is a New Skill Good?

Split into two measurable axes: effectiveness (better results) and economy (lower cost).

1. Pre-deploy ablation. Same task batch, two arms: A without skill, B with skill. Record success rate, human-fix count, token cost. net benefit = success_rate_lift × task_frequency − marginal_cost A skill that lifts success 30% but triggers 3×/year at +50k tokens each is a liability, not an asset.
2. Post-deploy shadow mode. New skill runs silent: produces suggestions but doesn't auto-execute; log "what would have happened". Watch adoption rate (accepted vs overridden). Adoption rate is the ultimate judge of "good", more honest than any offline metric. Boundary: write-operation skills (delete file, send mail) can only simulate/annotate in shadow, never really run.
3. Net-value card. One dashboard per skill: call count, success rate, adoption rate, avg tokens, cumulative net benefit. 90 days net-negative → suggest rewrite or retire. This is the consumer side of the depreciation table.

Judgment 3 — Does It Fit THIS User?

"Fits you" = triggers accurately + assumptions hold + style matches. All automatable:

1. Trigger precision (most affects feel, most skipped). Sample history positives (should trigger) and negatives (look-alike, shouldn't — "fix this bug" vs "explain what a bug is"). Replay → precision/recall. Low precision = annoying (over-triggers); low recall = wasted install. First manual label 10–20, then semi-auto, zero LLM cost.
2. Environment reconciliation (installed ≠ runs). Skill metadata declares deps: which tools / MCP / permissions / path habits / language, plus implicit model-capability assumptions (long context, code execution, multimodal). Reconcile against the user's real environment; missing → red flag. "Assumes macOS", "outputs Japanese", "needs 200k context" — invisible assumptions are the #1 reason a skill goes unused. This also feeds back into Judgment 1: a failed capability assumption = silent degradation.
3. Style baseline (lowest weight). Use the user's historical output + USER.md as reference; LLM-judge compares format, language, verbosity. Only used to break ties between equal-scoring skills.

Embedded Workflow (for "make a new skill")

When the user asks to create a skill, run the protocol inline before delivering:

1. Environment reconciliation — does the user's environment satisfy the skill's declared deps + capability assumptions?
2. Trigger replay — replay 10–20 sampled cases; report precision/recall.
3. Ablation — on a small task batch, measure lift vs cost.
4. Enter monitoring — on pass, register fingerprint + golden set; the resident console re-runs on each model bump.

Need-Based Suggestion (when no skill matches)

This is the "empty shelf" behavior: a user (especially one with zero skills/plugins) asks for something no installed skill covers. Don't just say "I can't" — surface the gap and offer to fill it.

1. Detect the need from the conversation: task type + domain (e.g. "quarterly financial report" → finance/analysis; "remove the AI tone" → humanizer; "build a dashboard" → frontend/chart).
2. Check the registry first — python scripts/console.py suggest --need "<need>". It returns either:
   • ✓ You already have an available skill/plugin: <name> → use it, don't install anything; or
   • 💡 Suggested new: [install:<slug>] <title> → no cover, recommend the category.
3. If uncovered, search the marketplace for real candidates (skills/plugins/experts) and present a card with evidence: what it does, why it fits this need, source. Never auto-install — the user picks.
4. On successful install or finding a matching skill, run the fitness evaluation from Judgment 3 (trigger precision, environment reconciliation, style match) to confirm fit.