shimo4228/agent-stocktake

agent-stocktake — Agent Definition Quality Audit

Evaluate every agent definition under <harness>/agents/*.md and assign each a verdict: Keep / Improve / Update / Merge / Demote to skill / Dissolve / Retire. The audit unit is the file, but the cost unit is split across two layers — that split is the reason this skill exists as a third sibling next to skill-stocktake and rules-stocktake.

Design note — the hybrid cost model. A skill's cost is trigger pollution (probabilistic firing degrades selection); a rule's cost is residency (always loaded). An agent has both at once: its description is injected into every session via the "Available agent types" listing (residency, like a rule), while its body loads only when the agent is invoked (invocation, like a skill — but triggered by Claude's delegation judgment, not by description matching against the user's words). So the description is audited on residency density and the body on invocation quality, with different questions for each layer.

Design note 2 — approved edits are applied by this skill itself, on the apply branch. Same reasoning as rules-stocktake: no improvement engine exists for agent definitions, and the corpus is small enough to read in one context — delegation would be overengineering. The handoff exception is Demote to skill (creating a skill is skill-writer's job).

Roots. <harness> below is the harness this audit covers: ~/.claude by default, or a repository's .claude/ when skill harness-stocktake runs in project mode (its Roots section maps the paths, names the evidence that is n/a there, and says which convention checks apply). The ledger <ledger> belongs to that harness: ${CLAUDE_SKILL_DIR}/results.json when this skill's directory is inside <harness> (the harness carries its own copy of the skill), else <harness>/stocktake/agent-stocktake.json (an installed plugin, or project mode). Either way it is committed with the harness's other changes on the apply branch.

Modes ($ARGUMENTS)

Argument	Behavior
none / full	Read and evaluate every agent definition (default)
changed	Re-evaluate only files whose mtime is newer than <ledger>'s evaluated_at; carry the rest forward from the ledger

changed detects changes inline (no script):

Paths below start at ${CLAUDE_SKILL_DIR}, the directory holding this SKILL.md; an agent that does not substitute the variable reads it as that directory.

find <harness>/agents -name "*.md" -newermt "$(jq -r .evaluated_at "<ledger>")"

As in rules-stocktake, the Phase 1 integrity checks always run over the full set — retiring a skill or hook silently breaks a reference inside an unmodified agent body, and mtime cannot see that. Any agent whose reference check (Step 3 below) newly fails joins the re-evaluation set. The script has no changed mode either, for a different reason: the near-duplicate pairs and the residency total are set properties, so a partial corpus would give wrong numbers rather than fewer ones.

Phase 1 — Evidence, inventory, usage

Step 1 — Run the evidence script (do not count by hand)

uv run --frozen --project "${CLAUDE_SKILL_DIR}" \
       --directory "${CLAUDE_SKILL_DIR}" \
       python scripts/agent_evidence.py --root <harness>

Evidence mode: JSON on stdout, exit 0 however many findings (exit 2 only when the corpus is unreadable). It measures and enumerates; it never assigns a verdict.

Transcribe the JSON into findings — never re-measure by eye. What it gives you:

JSON field	Replaces
desc_words / body_lines / total_desc_words	the wc -w / wc -l pass
tools.items[].status + registry.known_tools	builtin / unverified / mcp. unverified means "not in the script's dated list", not "does not exist" — you hold the live tool registry and the script does not, so confirm an unverified against your own tool list before calling it Update evidence. registry.known_tools.as_of is that list's date (null when --known-tools replaced it)
tools.items[].server_in_config + registry.mcp_config_files	Not a verdict. It says only whether the server appears in the local config files listed — servers supplied by a connector never appear there at all, so a false is not "retired". null means no config source parsed. Check the status of each source before using this at all; a false beside a non-ok source is not evidence
unreadable	files the scan could not open, with a reason. Non-empty means the scan is incomplete — name them and fix before stating any total, because a suppression instruction inside an unread file is now certified absent
tools_unparsable	agents whose tools: line did not parse (e.g. YAML flow style). Their items is [], which looks identical to an agent with no tools: key — i.e. unrestricted. It is not; the list was simply not read
description_near_duplicates	the near-identical-description check; twins split delegation traffic, so each pair feeds the Stage 1 overlap question
suppression_candidates	line-numbered candidates for the Stage 1 suppression question (the catalog is bilingual because the corpus is)
always_never_candidates	line-numbered candidates for the over-constraint question, body only — a description is a delegation trigger, not an instruction to the agent

The last two are candidates, not findings — read every cited line before writing it up. The catalog is also a floor, not a census: it holds five phrasings drawn from one corpus, so a suppression written some other way ("skip anything you're unsure about", "avoid findings that would just be noise") appears in no JSON field. Keep reading for those; an empty list is not a clean bill.

The measured false-positive ratios live outside this file, so this step carries no numbers that go stale.

Step 2 — Read the gates that already own the rest

Three checks stay outside the script because, in a harness that has a lint gate (<harness>/scripts/hooks/harness_lint.py), that gate owns them. Read its output when it exists; without it, make the three checks with Read / Grep:

• Frontmatter parses and carries name, description, origin, model, and model is an alias → the lint script's lint_agents (in the author's harness it runs in .claude/verify.sh and the pre-commit hook). A missing origin, where the harness states the origin convention, is still an integrity finding, feeding Improve.
• name equals the filename stem → the same lint_agents. It admits no judgment and the delegation registry keys on it, so it belongs to a gate that stops a commit — not to evidence a reader may or may not transcribe. A finding here is a blocked commit, not a stocktake verdict.
• Markdown links inside agent bodies resolve → the same script's lint_markdown_links (its scopes include agents).

Step 3 — Resolve bare-path references (the one check nothing else owns)

lint_markdown_links sees only link syntax, and skill-health's scan_refs.py scans the skills tree. A path written as prose inside an agent body (<harness>/hooks/foo.sh, skills/foo/SKILL.md) is covered by neither. Grep the corpus for them and ls each one.

Step 4 — Inventory

Enumerate with Glob: <harness>/agents/*.md. Read every file into one context (the corpus is small). The residency and body columns come from the Step 1 JSON — never trust figures written in docs, including this one.

Step 5 — Usage counts

Evidence input: low usage nominates a candidate, and the content decides the verdict. When the setup keeps an agent usage log (<harness>/metrics/agent-usage.jsonl; absent = usage n/a), read it inline (a usage-logging hook such as log-agent-usage.sh appends one invoke event per Agent-tool launch, keyed by its agent field, whose value is the subagent_type) and count per-agent events over 7 / 30 / 90 days. Aggregate with a throwaway python3/jq one-liner.

• If the log is missing or its first event is younger than 90 days, render usage as — (unmeasured). Never render it as 0 — unmeasured and unused are different facts.
• Counts are lower bounds: only Agent (Task) tool launches are captured. Workflow agent() workers, plugin-internal dispatch, and built-in machinery that bypasses the tool call do not reach the hook. A low-usage agent enters Stage 1 as a Retire/Dissolve candidate; its value can be episodic (e.g. paper reviewers fire only near a deposit), so the content decides.
• Log exists since 2026-07-27; before that date there is no measurement at all.

State the scan result up front, taking the counts from the Step 1 JSON: agents_total minus anything in unreadable, which must be named, total_desc_words (the per-session residency tax of the listing), integrity failures, and whether usage is measurable. Carry failures into Stage 1 as pre-computed evidence.

Phase 2 — Evaluation (fully inline, holistic)

Read every body while seeing the whole set.

Stage 1 — binary screen (every agent). Explicit Yes/No per item; surface only the No answers. The first two questions audit the description layer (residency), the rest the body layer (invocation):

• Description earns its residency? — dense, distinct, and selection-enabling in the always-loaded listing; states when to delegate AND when not to
• Description truthful to the body? — what it promises is what the body does (a drifted description misroutes delegation every session, even if the body is fine)
• Body free of suppression instructions? — start from this agent's suppression_candidates in the Phase 1 JSON, handled per Step 1. What counts: confidence thresholds ("only report findings you are ≥N% sure of", "assign each finding a confidence and discard the low ones"), severity floors ("only high-severity"), "be conservative" framings. The current-generation guidance is: report everything, filter in a separate pass — a suppression instruction is followed literally and silently drops findings. A No here is an Improve-by-inversion candidate: rewrite the instruction in the opposite direction, never just delete the section (deleting leaves the suppressive frame; inverting replaces it)
• External-origin copy current? — for an agent whose origin is an external repo and whose body is a verbatim copy (e.g. a built-in override such as Explore), the source version named in the body's leading comment matches the installed tool (claude --version); a mismatch is an Update (diff against the bundle and re-copy). The suppression and over-constraint questions are skipped for such copies — the body is the vendor's, not ours
• Body free of previous-generation over-constraint? — exhaustive step-by-step procedures for judgment the current model holds natively, repeated emphasis, ALWAYS/NEVER pairs that the surrounding-context judgment should own. Phase 1's always_never_candidates locates the tokens; deciding whether a hit is a directive or merely quotes the words is this question's job
• Not absorbed by the substrate? — does the harness now cover this agent's job natively (native review machinery, plan mode, built-in slash commands)? Absorption → Dissolve candidate; the claim must name its absorber concretely. Judge with the fresh/rich context axis: roles that gain from fresh context (review, adversarial verification, essence evaluation — decorrelation from the proposer's sunk cost) legitimately live in a subagent; roles that gain from rich context (planning, generation, implementation — user intent, in-conversation constraints) are main-loop work, so for them the main loop itself counts as an absorber. Two auxiliary rationales legitimately override the rich-context pull: a frozen-input render contract — the caller freezes a self-contained packet before invocation, so conversation context is not needed by design (prompt-forager, which gets one line on purpose) — and bulk context isolation — the work reads or produces volume that would pollute the main context (researcher, one web-research angle per run)
• Technical references current? — unconditionally verify every artifact the body names: ls each referenced path, run --help / version checks for named commands, flags, model names and tool lists. "Verify if it looks stale" is banned phrasing — the condition is what dilutes; deterministically checkable claims get deterministic checks, every time
• Unique within the set? — no other agent (or skill) owns the same job; a documented orchestrator→sub-agent split is NOT overlap

Eight questions and no more — further decomposition degrades holistic judgment (see References).

Stage 2 — verdict pressure-test (non-Keep candidates only). Generate 1–3 agent-specific atomic yes/no questions that try to refute the draft verdict, each answered with one line of evidence (file read, path check, --help, WebSearch, harness-doc check). For Dissolve candidates one question is mandatory: "Can the absorbing harness feature be named concretely — Yes/No" — an absorption claim that cannot name its absorber is refuted. And when a Dissolve is about to be refuted by a capability the substrate counterpart lacks (a tool, a wired sub-agent), one counter-question is mandatory before accepting the refutation: "Is the subagent the right place to use that capability — or does the main loop hold it anyway?" Capability existence is necessary but not sufficient; the fresh/rich context axis decides where the capability belongs (precedent: the planner refutation collapsed because the main loop holds the full Agent tool). Keep-bound agents get no dynamic questions.

Evaluation is holistic judgment, not a numeric rubric — binary answers are evidence, never aggregated into a score. Evaluation is origin-blind (vendored, self-written and customized agents all get the same checklist); where the harness states the origin convention, a missing origin header is itself a finding.

Aggregate residency cost (set-level): every description loads into every session, and the longer the listing, the weaker each entry's selection signal. The Keep bar rises with total description words — a merely-adequate agent is a Merge/Retire candidate on dilution grounds alone when the listing is crowded. A judgment input, never a quota.

Verdict	Meaning
Keep	Earns both layers: description dense and truthful, body current and unique
Improve	Worth keeping, needs tightening — includes inversion of suppression instructions (rewrite direction, don't delete)
Update	Referenced technology/tool/model is outdated (verified, with evidence)
Merge into [X]	Substantial overlap with another agent; name the target
Demote to skill	The value is the instructions, not the separate context/process — move to the skill layer via skill-writer
Dissolve	Absorbed by the substrate. Retirement by success — delete before the stale body overrides newer defaults; record the why in a decision-history entry
Retire	Defect-based removal: low quality, stale, broken beyond repair

Mandatory-surface rule: a No on the absorption question MUST surface the agent as a Dissolve candidate (final call is the user's) — an absorbed agent still receives delegation traffic and actively applies its stale body to current work.

Phase 3 — Summary

Render a table: Agent | Desc words | Body lines | Usage 90d | Verdict | Reason (Usage 90d is — while unmeasured, per Phase 1). Close with one line reporting total description words and the delta since the previous audit — input to the aggregate-residency judgment next run.

Phase 4 — Report and apply

Turn every verdict into items (source agents) of the report contract in skill: harness-stocktake and hand them to it: one report, one approval from the author, application on a task branch, a completion report, an ff-only merge. Called alone, this skill runs that contract's Phases 2–5 with its own items. Each item carries, by verdict:

• Improve / Update / Merge: change = the concrete edit as a diff; inversion edits show old and new direction side by side. Application edits the agent file directly on the branch (Design note 2)
• Demote to skill: change = the skill-writer intent packet and whether the agent file is deleted or reduced. Application creates the skill through skill-writer's report path
• Dissolve / Retire: reason = the absorption evidence or defect and what covers the need instead; impact = skills and rules that name this agent (grep the harness) and the public repo copy. Application records a Dissolve's why via decision-history
• Keep: reason only

Where skill harness-stocktake is not installed, show the same items as one table ordered Dissolve / Retire → Merge / Demote → Improve / Update → Keep, take one approval with the author's overrides, then apply.

• Update the ledger on the apply branch, in the same commit as the changes: Read <ledger> → merge verdicts (an item the author skipped keeps its verdict with "skipped": true) → Write back (evaluated_at = real UTC from date -u +%Y-%m-%dT%H:%M:%SZ). In changed mode, preserve prior verdicts of files not re-evaluated.
• Public-copy note: when the setup publishes copies of its agents elsewhere, editing or retiring one leaves that copy stale — point the user at their publish step.

Reason quality (required)

Every reason must be self-contained and decision-enabling. For non-Keep verdicts, cite the No answers (question + one-line evidence):

• Improve (inversion): Bad: "Has a threshold" / Good: "L23 'only report issues you are 80%+ confident in' suppresses findings per current-generation guidance — invert to 'report everything; caller filters in a separate pass'."
• Dissolve: name the absorber. Bad: "Not needed" / Good: "Harness plan mode + Plan agent type now provide native planning delegation; body duplicates and predates it. Record the why in decision-history, then delete."
• Merge: name the target + what to integrate.
• Keep (carry-forward in changed mode): restate the rationale.

Ledger (<ledger>, lean)

{
  "evaluated_at": "2026-07-26T00:00:00Z",
  "total_desc_words": 0,
  "agents": {
    "security-reviewer": {
      "path": "<harness>/agents/security-reviewer.md",
      "desc_words": 38,
      "body_lines": 120,
      "verdict": "Keep",
      "reason": "...",
      "mtime": "2026-07-01T00:00:00Z"
    }
  }
}

Created on the first run — do not pre-seed. Update inline with Read/Write, not a script.

Related

• skill-stocktake / rules-stocktake — the two siblings; this skill fuses their cost models (description = residency, body = invocation).
• generation-audit — the cross-asset orchestrator; on a model-generation change it collects runtime-layer evidence (conflict / redundancy) and /claude-api prompt-audit findings, and hands the agents slice to this skill as Stage 2 evidence.
• skill-writer (when installed) — handoff target for the skill-creation half of Demote.
• decision-history (when installed) — records the why of a Dissolve.
• harness-boundary — design-time lens (layer / portability / obsolescence) for proposed mechanisms; applied to an installed agent, its Delete / Move are Stage 2 evidence only.
• Usage measurement: a usage-logging hook writing <harness>/metrics/agent-usage.jsonl, when the setup has one (a measurement layer independent of stocktake).

References

The two-stage binary-question design (screen → verdict pressure-test, holistic verdict, no score aggregation) is inherited from skill-stocktake / rules-stocktake and follows the checklist-decomposition evaluation line: BinEval "Ask, Don't Judge" (arXiv:2606.27226), CheckEval (arXiv:2403.18771), TICK (arXiv:2410.03608) — over-decomposition degrades correlation on holistic quality, hence eight questions and no score. The suppression-instruction question implements the current-generation prompting guidance (report everything, filter in a separate pass); the absorption question and Dissolve verdict implement Scaffold Dissolution from the Agent Knowledge Cycle (inward / downward vectors + model-generation trigger).