Capability Depreciation Table (skill-auditor)
When the model is upgraded, installed skills / plugins won't throw errors, they will only quietly degrade. This skill turns "degradation" into measurable numbers.
Three Judgments, Each with Evidence
| User's Question | How to Determine | Key Prerequisite |
|---|---|---|
| Have capabilities degraded? | Re-run the same golden set on the new model; pass rate drop ≥20pp with consistent direction | Fingerprint unchanged. Fingerprint changed means it was 'modified', not degraded |
| Is the new skill/plugin useful? | Ablation study (with it / without it) + adoption rate, calculate net benefit | No control group means only "feels good" |
| Does it fit me? | Trigger precision (triggers when it should, doesn't trigger when it shouldn't) + environment reconciliation + style baseline | Use real historical prompts, not imagined ones |
How to Invoke
When the plugin has registered tools, use the tools directly (this is the preferred method):
skill_audit_status— Depreciation table: fingerprint / baseline pass rate / risk / judgmentskill_audit_recommend— Strong / Watch / Unverified / Degraded / Prune-candidate, five tiersskill_audit_check— Whether the current model has changed relative to the last regression baselineskill_audit_scan— Re-discover assets after installing new itemsskill_audit_suggest— What to add when the user's need is not covered by any asset
Without a tool surface (e.g., hosts that only have skills installed), use the CLI:
python <data-dir>/scripts/console.py --host deepseek status
python <data-dir>/scripts/console.py --host deepseek recommend
python <data-dir>/scripts/console.py --host workbuddy status --root ~/.workbuddy-ai/skills
<data-dir> defaults to ~/.dsh/skill-auditor (can be overridden with SKILL_AUDITOR_HOME). The engine is pure standard library Python 3.8+, with no third-party dependencies.
When the User Has No Available Skills / Plugins
This is the skill's proactive mechanism, do not wait for the user to ask:
- First check the installed registry — if there is coverage, do not suggest a new one (avoid duplicate installations);
- Truly none available →
skill_audit_suggest --need "<one-sentence need>"; - Output "suggested addition + search term", installation is confirmed by the user, no automatic installation.
Hard Constraints (Must State Honestly, Do Not Package as Conclusions)
- A baseline with K=1 is self-assessment, not a statistical conclusion. To render a 'degradation' verdict, K≥3 is required.
- The golden set may contain synthetic samples (when the user's history is insufficient). Synthetic samples can test 'is the structure correct', but cannot test 'does it resemble you'. Such samples are marked with
"synthetic": truein the file. riskis an indicator of the maintenance surface (lines of code + tool dependencies), it does not represent value. High risk does not mean it should be deleted.- Removal and installation must be confirmed; removal prioritizes 'disable / archive', no silent deletion.
- Degradation attribution has only four categories, do not expand: Tool call error → fix tool description; Output structure correct but content wrong → rewrite instructions; Should have triggered but didn't → modify trigger description; All intermediate steps correct but final state wrong → only then might be a model capability regression.
Post-Judgment Actions
| Judgment | Action |
|---|---|
| strong | Keep, can be pinned as recommended |
| watch | Keep, prioritize re-running on next model change |
| unverified | First build a golden set (3–5 real prompts) then judge, do not deprecate for now |
| degraded | First attribute, then decide to rewrite or disable; do not delete directly |
| prune-candidate | No baseline + very high maintenance surface → suggest evaluation, do not auto-delete |