harness-retro

Harness retro

Find where past sessions went wrong and change the setup so it does not happen again. Every finding must point to evidence in a transcript. Never apply a change before the user approves it.

1. Mine the transcripts

The miner is a stdlib-only script next to this file. Run it with -I. Write its output to the session scratchpad, or a temp directory if there is none:

python3 -I <this skill's base directory>/scripts/mine.py [PATH] [--since 30d] --out <scratchpad>/retro.json

• With no PATH it mines the project for the current working directory. To mine a different project, give its working directory path, or use --project=<key> with the folder name under ~/.claude/projects. Use the = form, because keys start with -.
• Use --since if the user named a window. Raise --max-bytes (default 40000) only if the output is truncated and you need more.
• If the user asks about every project, run it once per folder in ~/.claude/projects and look at the projects with the most friction.

The script prints a one-line summary. If it scanned few or no sessions, tell the user. Claude Code deletes transcripts after cleanupPeriodDays (default 30), so older history may be gone. Suggest raising that setting if they want deeper retros later.

2. Read the hits

The JSON lists sessions ranked by friction score. Each hit has signal, score, ts, prior_prompt (what the user asked), recent_actions (Claude's last tool calls before the hit), and either user_reaction or detail.

Signals, roughly in order of value:

• user_correction: the user pushed back. Scores of 5 or more mean strong frustration or a repeat ("still", "again", "I said", swearing).
• edit_revert and edit_churn: Claude went back and forth on the same file.
• tool_error_streak: Claude kept hitting the same failure.
• interrupted: the user stopped Claude mid-turn. This usually means it was heading the wrong way.
• permission_denied: Claude tried something the user or the classifier blocked.
• self_retraction: Claude admitted an error. This signal is noisy, so use it only as supporting evidence.

The regexes are crude. Throw away false positives, such as "don't" in a normal instruction or "try again" after a network error. When a hit is ambiguous, open the transcript around it. The file is ~/.claude/projects/<key>/<session>.jsonl; grep for the timestamp and read a few lines either side. Don't read whole transcripts.

3. Group into patterns

Group the real hits into recurring patterns. A pattern is one root cause, written as something Claude does wrong. Examples: "Generates new images instead of using the approved asset library", "Edits the current version instead of cutting a new one", "Claims a change is visible without checking the built output". For each pattern, record:

• How often it happened, and in how many sessions.
• Two or three short quotes from the user, with session title and date.
• The root cause. Pick one: missing knowledge, a rule that existed but was ignored, a bad default workflow, missing verification, or a tool or permission gap.

A single incident counts only if the user was clearly angry or the cost was high, for example destructive or lost work.

4. Check the current harness

Before proposing anything, read what is already in place:

• ~/.claude/CLAUDE.md, plus the project CLAUDE.md and any nested ones.
• ~/.claude/settings.json, and the project's .claude/settings.json and .claude/settings.local.json (hooks, permissions).
• The project memory dir ~/.claude/projects/<key>/memory/ and its MEMORY.md.
• Skills in ~/.claude/skills/ and the project's .claude/skills/.

If a pattern is already covered by a rule that Claude ignored, adding the same rule again won't help. Make the rule more specific and put it where it will be seen, or move to a stronger mechanism.

5. Choose the fix

For each pattern, use the lightest mechanism that will actually work:

Root cause	Fix
A project fact Claude didn't know	A memory file in the project memory dir, with frontmatter name, description, metadata.type, a body with Why: and How to apply:, and a one-line pointer in MEMORY.md. Update an existing memory rather than duplicating it.
A standing rule for this repo	Project CLAUDE.md
A rule for every project	~/.claude/CLAUDE.md
A rule that keeps being broken and can be checked by a script (formatting, banned strings, a required command before commit)	A hook in settings.json. Use the update-config skill if it is available.
Repeated permission prompts or denials for safe commands	Propose allowlist entries, or suggest the fewer-permission-prompts skill if it is available. Don't widen permissions for destructive commands.
A multi-step workflow that keeps going wrong	A new or edited skill (use skill-creator if it is available).
A Claude Code product bug	Note it in the report. There is no harness fix.

Write rules as concrete, checkable instructions that name the specific thing, such as files, asset paths, commands or version naming. "Be careful with images" is too vague to act on.

6. Report and get approval

Present one section per pattern, most costly first. Each section gives:

1. The pattern, in one line.
2. The evidence: count, sessions, and 2 or 3 short quotes with date.
3. The root cause.
4. The proposed change: the target file and the exact text or diff.

Then ask which changes to apply. A multiSelect AskUserQuestion with one option per pattern works well. Apply only the approved changes, then list what changed and where.

Writing rules

Anything written into CLAUDE.md, memory or skills must follow any writing rules in the user's own CLAUDE.md files, such as tone, banned words or attribution rules. Keep it short and specific. Quotes from transcripts can include secrets the user pasted. The miner redacts common key formats, but check before you copy text into any file.