shimo4228/agent-stocktake

agent-stocktake — Agent Definition Quality Audit

Evaluate every agent definition under ~/.claude/agents/*.md and assign each a verdict: Keep / Improve / Update / Merge / Demote to skill / Dissolve / Retire. The audit unit is the file, but the cost unit is split across two layers — that split is the reason this skill exists as a third sibling next to skill-stocktake and rules-stocktake.

Design note — the hybrid cost model. A skill's cost is trigger pollution (probabilistic firing degrades selection); a rule's cost is residency (always loaded). An agent has both at once: its description is injected into every session via the "Available agent types" listing (residency, like a rule), while its body loads only when the agent is invoked (invocation, like a skill — but triggered by Claude's delegation judgment, not by description matching against the user's words). So the description is audited on residency density and the body on invocation quality, with different questions for each layer.

Design note 2 — edits are applied in-session. Same reasoning as rules-stocktake: no improvement engine exists for agent definitions, and the corpus is ~20 small files — delegation would be overengineering. The handoff exception is Demote to skill (creating a skill is skill-creator's job).

Modes ($ARGUMENTS)

Argument	Behavior
none / full	Read and evaluate every agent definition (default)
changed	Re-evaluate only files whose mtime is newer than results.json's evaluated_at; carry the rest forward from the ledger

changed detects changes inline (no script):

find ~/.claude/agents -name "*.md" -newermt "$(jq -r .evaluated_at ~/.claude/skills/agent-stocktake/results.json)"

As in rules-stocktake, the Phase 1 integrity checks always run over the full set — retiring a skill or hook silently breaks a reference inside an unmodified agent body, and mtime cannot see that. Any agent whose reference check (Step 3 below) newly fails joins the re-evaluation set. The script has no changed mode either, for a different reason: the near-duplicate pairs and the residency total are set properties, so a partial corpus would give wrong numbers rather than fewer ones.

Phase 1 — Evidence, inventory, usage

Step 1 — Run the evidence script (do not count by hand)

uv run --project ~/.claude/skills/agent-stocktake \
       --directory ~/.claude/skills/agent-stocktake \
       python scripts/agent_evidence.py --root ~/.claude

Evidence mode: JSON on stdout, exit 0 however many findings (exit 2 only when the corpus is unreadable). It measures and enumerates; it never assigns a verdict.

Transcribe the JSON into findings — never re-measure by eye. What it gives you:

JSON field	Replaces
desc_words / body_lines / total_desc_words	the wc -w / wc -l pass
tools.items[].status + registry.known_tools	builtin / unverified / mcp. unverified means "not in the script's dated list", not "does not exist" — you hold the live tool registry and the script does not, so confirm an unverified against your own tool list before calling it Update evidence. registry.known_tools.as_of is that list's date (null when --known-tools replaced it)
tools.items[].server_in_config + registry.mcp_config_files	Not a verdict. It says only whether the server appears in the local config files listed — servers supplied by a connector never appear there at all, so a false is not "retired". null means no config source parsed. Check the status of each source before using this at all; a false beside a non-ok source is not evidence
unreadable	files the scan could not open, with a reason. Non-empty means the scan is incomplete — name them and fix before stating any total, because a suppression instruction inside an unread file is now certified absent
tools_unparsable	agents whose tools: line did not parse (e.g. YAML flow style). Their items is [], which looks identical to an agent with no tools: key — i.e. unrestricted. It is not; the list was simply not read
description_near_duplicates	the near-identical-description check; twins split delegation traffic, so each pair feeds the Stage 1 overlap question
suppression_candidates	line-numbered candidates for the Stage 1 suppression question (the catalog is bilingual because the corpus is)
always_never_candidates	line-numbered candidates for the over-constraint question, body only — a description is a delegation trigger, not an instruction to the agent

The last two are candidates, not findings, and roughly one in five is real — read every cited line before writing it up. The catalog is also a floor, not a census: it holds five phrasings drawn from one corpus, so a suppression written some other way ("skip anything you're unsure about", 「ノイズになる指摘は避ける」) appears in no JSON field. Keep reading for those; an empty list is not a clean bill.

For the measured false-positive ratios and which lines produced them, see ADR-0054 — they are recorded once, there, so this step does not carry numbers that go stale.

Step 2 — Read the gates that already own the rest

Three checks stay outside the script because another gate owns them. Read their output; do not re-implement:

• Frontmatter parses and carries name, description, origin, model, and model is an alias → scripts/hooks/harness_lint.py lint_agents, which runs in .claude/verify.sh and the pre-commit hook. A missing origin is still an integrity finding per rules/common/skills.md, feeding Improve.
• name equals the filename stem → the same lint_agents (moved there from this script 2026-08-27, RFC-0014). It admits no judgment and the delegation registry keys on it, so it belongs to a gate that stops a commit — not to evidence a reader may or may not transcribe. A finding here is a blocked commit, not a stocktake verdict.
• Markdown links inside agent bodies resolve → the same script's lint_markdown_links (its scopes include agents).

Step 3 — Resolve bare-path references (the one check nothing else owns)

lint_markdown_links sees only link syntax, and skill-health's scan_refs.py scans the skills tree. A path written as prose inside an agent body (~/.claude/hooks/foo.sh, skills/foo/SKILL.md) is covered by neither. Grep the corpus for them and ls each one. This is the check the changed-mode rule above depends on, so it runs over the full set even in changed mode.

Step 4 — Inventory

Enumerate with Glob: ~/.claude/agents/*.md. Read every file into one context (the corpus is small). The residency and body columns come from the Step 1 JSON — never trust figures written in docs, including this one.

Step 5 — Usage counts

Evidence input, never a verdict trigger. Read ~/.claude/metrics/agent-usage.jsonl inline (the hook log-agent-usage.sh appends one invoke event per Agent-tool launch, keyed by subagent_type) and count per-agent events over 7 / 30 / 90 days. Aggregate with a throwaway python3/jq one-liner.

• If the log is missing or its first event is younger than 90 days, render usage as — (unmeasured). Never render it as 0 — unmeasured and unused are different facts.
• Counts are lower bounds: only Agent (Task) tool launches are captured. Workflow agent() workers, plugin-internal dispatch, and built-in machinery that bypasses the tool call do not reach the hook. Never Retire/Dissolve on low usage alone — an agent's value can be episodic (e.g. paper reviewers fire only near a deposit).
• Log exists since 2026-07-27; before that date there is no measurement at all.

State the scan result up front, taking the counts from the Step 1 JSON: agents_total minus anything in unreadable, which must be named, total_desc_words (the per-session residency tax of the listing), integrity failures, and whether usage is measurable. Carry failures into Stage 1 as pre-computed evidence.

Phase 2 — Evaluation (fully inline, holistic)

Read every body while seeing the whole set.

Stage 1 — binary screen (every agent). Explicit Yes/No per item; surface only the No answers. The first two questions audit the description layer (residency), the rest the body layer (invocation):

• Description earns its residency? — dense, distinct, and selection-enabling in the always-loaded listing; states when to delegate AND when not to
• Description truthful to the body? — what it promises is what the body does (a drifted description misroutes delegation every session, even if the body is fine)
• Body free of suppression instructions? — start from this agent's suppression_candidates in the Phase 1 JSON, read each cited line, then keep reading for phrasings the catalog does not know (it is a floor, not a census). What counts: confidence thresholds ("only report findings you are ≥N% sure of", 「確信度を付け、低いものは捨てる」), severity floors ("only high-severity"), "be conservative" framings. The current-generation guidance is: report everything, filter in a separate pass — a suppression instruction is followed literally and silently drops findings. A No here is an Improve-by-inversion candidate: rewrite the instruction in the opposite direction, never just delete the section (deleting leaves the suppressive frame; inverting replaces it)
• Body free of previous-generation over-constraint? — exhaustive step-by-step procedures for judgment the current model holds natively, repeated emphasis, ALWAYS/NEVER pairs that the surrounding-context judgment should own. Phase 1's always_never_candidates locates the tokens; deciding whether a hit is a directive or merely quotes the words is this question's job
• Not absorbed by the substrate? — does the harness now cover this agent's job natively (native review machinery, plan mode, built-in slash commands)? Absorption → Dissolve candidate; the claim must name its absorber concretely. Judge with the fresh/rich context axis (ADR-0023): roles that gain from fresh context (review, adversarial verification, essence evaluation — decorrelation from the proposer's sunk cost) legitimately live in a subagent; roles that gain from rich context (planning, generation, implementation — user intent, in-conversation constraints) are main-loop work, so for them the main loop itself counts as an absorber. Two auxiliary rationales legitimately override the rich-context pull (ADR-0024): a frozen-input render contract — the caller freezes a self-contained packet before invocation, so conversation context is not needed by design (adr-writer per ADR-0016, prompt-writer; likewise repo-grounded work whose input is the codebase, not the conversation — codemap-writer, scout) — and bulk context isolation — the work reads or produces volume that would pollute the main context (e2e-runner, refactor-cleaner)
• Technical references current? — commands, flags, model names, tool lists (verify with --help / WebSearch when they look stale)
• Unique within the set? — no other agent (or skill) owns the same job; a documented orchestrator→sub-agent split is NOT overlap

Seven questions and no more — further decomposition degrades holistic judgment (see References).

Stage 2 — verdict pressure-test (non-Keep candidates only). Generate 1–3 agent-specific atomic yes/no questions that try to refute the draft verdict, each answered with one line of evidence (file read, path check, --help, WebSearch, harness-doc check). For Dissolve candidates one question is mandatory: "Can the absorbing harness feature be named concretely — Yes/No" — an absorption claim that cannot name its absorber is refuted. And when a Dissolve is about to be refuted by a capability the substrate counterpart lacks (a tool, a wired sub-agent), one counter-question is mandatory before accepting the refutation: "Is the subagent the right place to use that capability — or does the main loop hold it anyway?" Capability existence is necessary but not sufficient; the fresh/rich context axis decides where the capability belongs (precedent: planner's Agent(scout) refutation collapsed because the main loop holds the full Agent tool, ADR-0023). Keep-bound agents get no dynamic questions.

Evaluation is holistic judgment, not a numeric rubric — binary answers are evidence, never aggregated into a score. Evaluation is origin-blind (ECC / shimo4228 / customized all get the same checklist); a missing origin header is itself a finding.

Aggregate residency cost (set-level): every description loads into every session, and the longer the listing, the weaker each entry's selection signal. The Keep bar rises with total description words — a merely-adequate agent is a Merge/Retire candidate on dilution grounds alone when the listing is crowded. A judgment input, never a quota.

Verdict	Meaning
Keep	Earns both layers: description dense and truthful, body current and unique
Improve	Worth keeping, needs tightening — includes inversion of suppression instructions (rewrite direction, don't delete)
Update	Referenced technology/tool/model is outdated (verified, with evidence)
Merge into [X]	Substantial overlap with another agent; name the target
Demote to skill	The value is the instructions, not the separate context/process — move to the skill layer via skill-creator
Dissolve	Absorbed by the substrate. Retirement by success — delete before the stale body overrides newer defaults; record the why in an ADR
Retire	Defect-based removal: low quality, stale, broken beyond repair

Mandatory-surface rule: a No on the absorption question MUST surface the agent as a Dissolve candidate (final call is the user's) — an absorbed agent still receives delegation traffic and actively applies its stale body to current work.

Phase 3 — Summary

Render a table: Agent | Desc words | Body lines | Usage 90d | Verdict | Reason (Usage 90d is — while unmeasured, per Phase 1). Close with one line reporting total description words and the delta since the previous audit — input to the aggregate-residency judgment next run.

Phase 4 — Consolidation

Confirm one by one (config-gc's confirm-each design): walk the non-Keep candidates sequentially — evidence first, then [y/n/skip]. Never batch the approval; one agent, one decision. skip records the verdict unactioned.

• Improve / Update / Merge: present the concrete edit → on approval, apply it directly in this session (Design note 2). Inversion edits show old and new direction side by side.
• Demote to skill: hand skill creation to skill-creator, then delete or reduce the agent file per the user's call.
• Dissolve / Retire: per file, present (1) the absorption evidence or defect, (2) what covers the need instead, (3) removal impact — skills and rules that name this agent (grep the harness) and the public repo copy. Act only after the user confirms. For Dissolve, offer to record the why via adr-writer.
• Update the ledger: Read results.json → merge verdicts → Write back (evaluated_at = real UTC from date -u +%Y-%m-%dT%H:%M:%SZ). In changed mode, preserve prior verdicts of files not re-evaluated.
• Public-repo note: editing or retiring an origin: shimo4228 agent leaves the public repo stale — point the user at harness-sync.

Reason quality (required)

Every reason must be self-contained and decision-enabling. For non-Keep verdicts, cite the No answers (question + one-line evidence):

• Improve (inversion): Bad: "Has a threshold" / Good: "L23 'only report issues you are 80%+ confident in' suppresses findings per current-generation guidance — invert to 'report everything; caller filters in a separate pass'."
• Dissolve: name the absorber. Bad: "Not needed" / Good: "Harness plan mode + Plan agent type now provide native planning delegation; body duplicates and predates it. ADR the why, then delete."
• Merge: name the target + what to integrate.
• Keep (carry-forward in changed mode): restate the rationale.

results.json (lean ledger)

{
  "evaluated_at": "2026-07-26T00:00:00Z",
  "total_desc_words": 0,
  "agents": {
    "security-reviewer": {
      "path": "~/.claude/agents/security-reviewer.md",
      "desc_words": 38,
      "body_lines": 120,
      "verdict": "Keep",
      "reason": "...",
      "mtime": "2026-07-01T00:00:00Z"
    }
  }
}

Created on the first run — do not pre-seed. Update inline with Read/Write, not a script.

Related

• skill-stocktake / rules-stocktake — the two siblings; this skill fuses their cost models (description = residency, body = invocation).
• generation-audit — the cross-asset orchestrator; on a model-generation change it collects runtime-layer evidence (conflict / redundancy / drift classification) and hands the agents slice to this skill as Stage 2 evidence.
• skill-creator — handoff target for the skill-creation half of Demote.
• adr-writer — records the why of a Dissolve.
• config-gc — whole-config GC; this skill judges agent quality.
• harness-sync — syncs surviving origin: shimo4228 agents to the public repo.
• harness-boundary — design-time lens (layer / portability / obsolescence) for proposed mechanisms; applied to an installed agent, its Delete / Move are Stage 2 evidence only.
• Usage measurement: ~/.claude/hooks/log-agent-usage.sh → ~/.claude/metrics/agent-usage.jsonl (a measurement layer independent of stocktake, mirroring skill-stocktake's log-skill-usage.sh).

References

The two-stage binary-question design (screen → verdict pressure-test, holistic verdict, no score aggregation) is inherited from skill-stocktake / rules-stocktake and follows the checklist-decomposition evaluation line: BinEval "Ask, Don't Judge" (arXiv:2606.27226), CheckEval (arXiv:2403.18771), TICK (arXiv:2410.03608) — over-decomposition degrades correlation on holistic quality, hence seven questions and no score. The suppression-instruction question implements the current-generation prompting guidance (report everything, filter in a separate pass); the absorption question and Dissolve verdict implement rules/common/akc-cycle.md's Scaffold Dissolution (inward / downward vectors + model-generation trigger, ADR-0018).