akontadakis/whetstone-mcp

Whetstone

Measure how a weak model and a strong model perform on one MCP server, then close the gap by changing the server. You drive a Python harness that does every mechanical part — running sessions, scoring them, checking regressions, rendering reports. Your job is the judgement between those calls: writing tasks, reading failures, proposing diffs, and running the four user gates.

Invoking the harness

The harness lives at skills/whetstone-mcp/harness (call that $HARNESS). It is a uv project, and whetstone_mcp is importable only when $HARNESS is on PYTHONPATH. Write each step as a small script under <target>/.whetstone-mcp/scripts/ and run it with:

PYTHONPATH=$HARNESS uv run --project $HARNESS python <script>.py

Short one-offs can use cd $HARNESS && uv run python -c '...' instead. The harness environment also carries the official MCP SDK (mcp), so a script can import ClientSession, StdioServerParameters, mcp.client.stdio.stdio_client and mcp.client.streamable_http.streamablehttp_client directly when it needs to talk to the target server itself. Prefer whetstone_mcp.introspect.open_session over hand-rolling either transport: it picks stdio or local HTTP from the same arguments introspect takes. It yields a pair, not a session — unpack it as async with open_session(...) as (session, init), where session is an initialized ClientSession and init is the initialize() result.

All state for a target lives in <target>/.whetstone-mcp/: config.yaml, surface.json, tasks/ (one YAML per scored task, with the held-out ones in tasks/holdout/), references.json, corpus.json, state.json, bench/, manifests/, scripts/, reports/. The snippets below call that directory state and its manifest subdirectory manifests:

from pathlib import Path

state = Path("<target>") / ".whetstone-mcp"
manifests = state / "manifests"
state.mkdir(parents=True, exist_ok=True)
(state / "reports").mkdir(exist_ok=True)
manifests.mkdir(exist_ok=True)          # run_phase also creates it

Every harness name in this document imports from whetstone_mcp.<module> — config, tasks, transcript, scoring, introspect, bench, runner, guardrails, changes, report, freeze, pipeline, process, state, calibration.

The sandbox

sandbox/ is a deliberately bad MCP server carrying one planted defect per polish class, with a twelve-task ruler and an answer key in sandbox/DEFECTS.md. It is not part of any user's optimization — it is how this skill is regression-tested, and how you can see a whole round end to end without spending a real budget. sandbox/README.md has both run modes: a free deterministic one that is part of the harness test suite, and a real-model one you drive by hand. Read the answer key only after a run, never before.

Read references/model-traps.md before the first phase — it governs how you run each phase and how you are allowed to describe every number you report.

Rate-limit pauses (Steps 4, 5, 6 and 7 — and the Step 2 preflight)

Every call that spawns sessions — execute_run, run_phase, baseline — can raise RateLimitPause from whetstone_mcp.runner. It is not a failure to retry: spec §11 pauses the phase instead of burning the window on refusals. Wrap each of those calls. The shape, shown here around the Step 5 call it will most often fire on (cfg, tasks and bench are the names Steps 1-3 define):

from whetstone_mcp.runner import RateLimitPause
try:
    results = baseline(cfg, tasks, bench, manifests / "baseline.jsonl",
                       dict(os.environ))
except RateLimitPause as pause:
    print(f"PAUSED on run {pause.run_id}. "
          f"Resume time stated by the API: {pause.resume_time or 'none given'}")
    raise SystemExit(0)

When it fires, stop the step, tell the user the phase is paused, name pause.run_id and pause.resume_time (say "no resume time was stated" when it is None), and tell them that re-running this same step after the window reopens resumes from the manifest — the paused run wrote no manifest row and spent no attempt, so nothing is lost and nothing is double-counted. Never re-run in a loop to wait it out.

Resume skips a run only when its last manifest row says complete. A run recorded infra_failed is re-attempted the next time you invoke the same step with the same manifest path, and its newer row supersedes the old one — load_manifest_records returns the last row per run id.

A manifest-backed run cannot pause forever. Consecutive pauses on the same run id are counted in <manifest>.jsonl.pauses.json beside the manifest and capped at runner.PAUSE_CAP (2). On the third invocation that run is classified infra_failed instead of pausing — the matched line is kept in that workspace's stderr.log — and the phase moves on. The count clears the moment the run records any row. So if a step pauses twice and then reports an infra_failed run whose stderr.log names a rate limit, the signature was matching something that is not a refusal you can wait out: read that line before re-running, and say so to the user rather than treating it as a capability result. The cross-invocation counter belongs to run_phase (and therefore baseline); do not use bare execute_run for a resumable preflight or phase, because it has no manifest sidecar in which to persist that count.

Round state (state.json) — the only memory across sessions

An optimization runs for hours and survives interruptions, rate-limit pauses, and new sessions. Use the validated, atomic state API; do not recreate it in a snippet or keep a gate result only in conversation memory.

import asyncio, json, os
from pathlib import Path
from whetstone_mcp.bench import audit_bench, setup_bench
from whetstone_mcp.calibration import record_bundle
from whetstone_mcp.config import Config, load_config
from whetstone_mcp.introspect import introspect, Surface, Tool
from whetstone_mcp.pipeline import (
    baseline, load_manifest_records, load_record_transcript,
    merge_partition_results,
)
from whetstone_mcp.report import render_final_report
from whetstone_mcp.runner import RunSpec, run_phase
from whetstone_mcp.state import (
    atomic_write_json, checkpoint, load_state, model_results_from_rows,
    model_results_to_rows, proposals_from_rows, proposals_to_rows,
)
from whetstone_mcp.tasks import dump_task, load_task_set

state = (Path("<target>") / ".whetstone-mcp").resolve()
target_root = state.parent
repo = (target_root if (target_root / ".git").exists()
        else state / "worktree")
state_path = state / "state.json"
manifests = state / "manifests"
cfg = load_config(state / "config.yaml")
raw_surface = json.loads((state / "surface.json").read_text())
surface = Surface(raw_surface.get("instructions"),
                  [Tool(**t) for t in raw_surface["tools"]],
                  raw_surface.get("capabilities"))
bench = setup_bench(cfg, surface, state / "bench")
audit_bench(bench)

tasks_dir, holdout_dir = state / "tasks", state / "tasks" / "holdout"
tasks = load_task_set(tasks_dir, holdout_dir,
                      cfg.scored_count, cfg.holdout_count)
scored_tasks = [t for t in tasks if not t.holdout]
holdout_tasks = [t for t in tasks if t.holdout]
by_id = {t.id: t for t in tasks}

workflow = load_state(state_path)
before = model_results_from_rows(workflow.before)
results = model_results_from_rows(workflow.latest)
shown = proposals_from_rows(workflow.shown_proposals)
approved = proposals_from_rows(workflow.approved_proposals)
applied = [tuple(item) for item in workflow.applied]
remaining_findings = list(workflow.remaining_findings)
plateau = list(workflow.plateau)
calibration = list(workflow.calibration)
round_n = workflow.round_n
REF_EFFORT = workflow.ref_effort

WorkflowState rejects unknown or malformed fields and checkpoint rejects impossible stage jumps. Save at each boundary below, before doing the next action:

stage	checkpoint is written when	resume action
report_ready	the report and full proposal diffs have been shown	wait for Gate 2; never regenerate proposals
approved	the exact selected proposals and their base_commit have been approved	open the branch and apply that saved set
branch_opened	the idempotent round branch is at the approved base	apply the saved bundle once
applied	apply_bundle returned its commit map	build and run guardrails
guardrails_passed	build, schemas, baseline args, replay, and tests are green	re-introspect, reclassify additions, rebuild and preflight
rearmed	new surface, safety labels, bench, preflight, and corpus are durable	bump tasks if required, otherwise schedule the rerun
bump_pending	approved routes and pre-edit task YAML are durable	restore the snapshot and apply each bump exactly once
tasks_bumped	version/route edits and bump_routes are on disk	schedule forced-route reference verification
reverify_pending	its manifest path and allowed new route are saved	resume that same reference manifest
rerun_pending	held-out/scored manifest paths are saved	resume those exact manifests; do not reapply or rebump
revert_pending	the rejection evidence is durable, before any Git revert	idempotently finish revert, rebuild, restore artifacts, and append outcome
rejected	rollback/outcome work is durable, or Gate 2 approved none	checkpoint complete; never start a round directly from rejected
complete	results/report/state for the round are durable	start the next round or Step 7

The stage is the dispatcher at the top of Step 6. A cold session does only the action named in the last column; it never increments round_n, opens another branch, reapplies commits, or reruns a task bump just because the Python process restarted. surface.json must be written as asdict(surface) so this reload recreates the same bench.

At the top of the dispatcher, close the tiny final rejection checkpoint window:

if workflow.stage == "rejected":
    workflow = checkpoint(state_path, workflow, "complete")

surface.json must therefore be written as asdict(surface) — see Step 2 — or this reload cannot rebuild the bench, and a bench rebuilt from a different surface would carry a different allow list than the phase it is resuming.

Step 1 — Setup (done when load_config returns a Config)

Ask the user these questions in one conversation:

1. Models — the ids or aliases to compare, passed to claude -p --model. List them weakest first, strongest last. Then ask for a separate cheapest-first order for the same ids. Capability and price are independent: reference_model and ceiling ties use strength order; a target tie uses model_cost_order. A list in the wrong order freezes golden references with the wrong model or chooses an unnecessarily expensive target.
2. Server launch — either the command and working directory that start the server over stdio, or the URL of a local HTTP server the user already runs (spec §2 puts both in v1; remote servers with third-party auth are out of scope, so decline a URL that is not the user's own local server). Ask in the same breath which environment variables the server needs to run — a database path, a fixture directory, a feature flag. A plain value (a path, a flag) goes into server_env literally. A credential never does: write it as the reference ${NAME} and have the user export NAME in the shell that runs the steps. Spec §4: generated configs reference credentials only by variable name, never by value — and config.yaml lives inside the user's repo, one git add from their history. introspect and replay_corpus resolve ${NAME} from the environment at use time and raise IntrospectError naming the variable when it is unset; bench.mcp.json passes the reference through unexpanded, because Claude Code expands it itself. Say to the user, either way, that this server instance should hold fixture or test data, not production credentials.
3. Test command — how to run the target's own test suite. none is a valid answer; it changes what Step 6 may apply without extra consent. 3.5 Build command — how the server is built, when it runs from a build artifact (a node dist/, a wheel, a compiled binary) rather than from source. none is a valid answer for a server that runs from source. Step 6 runs it between the apply and the guardrails; without it a round measures the unchanged server and reads "no effect".
4. State reset — a command that returns the server to a known state between runs, or stateless.
5. Environment — confirmation that this server instance touches no production data and causes no external side effects.

Write the answers to <target>/.whetstone-mcp/config.yaml as models, model_cost_order, server_command + server_cwd (stdio) or server_url (local HTTP), server_env, test_command, build_command, reset_command, and optionally reference_model (it defaults to the last entry in models). load_config requires exactly one of server_command / server_url, and server_cwd alongside a command. In URL mode server_cwd is optional — the harness spawns nothing — but it is still where reset_command, build_command and test_command run, so load_config requires it as soon as any of those three is set and never falls back to the process's working directory: one config file must not mean different directories depending on where a script was started.

models: [haiku, opus, fable]          # weakest first, strongest last
model_cost_order: [haiku, fable, opus] # cheapest first; every model exactly once
server_command: ["uv", "run", "server.py"]
server_cwd: /abs/path/to/target
server_env:
  TOY_FIXTURE_DIR: /abs/path/to/fixtures   # plain value: literal
  TOY_API_TOKEN: ${TOY_API_TOKEN}          # credential: reference only
test_command: "uv run pytest"
build_command: null                   # or e.g. "npm run build"
reset_command: null              # required if any tool is stateful_safe
reset_timeout_s: 60

Both kinds of id end up in a run id, and from there in a workspace directory name, so both are validated — but not to the same strictness, because they are not the same kind of thing.

Task ids are yours to choose, so they must be plain path segments: ^[A-Za-z0-9][A-Za-z0-9._-]*$, enforced when a Task is built or loaded.

Model ids are the provider's to choose, so load_config refuses only what actually endangers a path or a log line — a separator (/, \), whitespace, a control character, the empty string, . and ... Everything else is accepted, including a provider-qualified id like us.anthropic.claude-opus-4-5-v1:0: the harness sanitizes the workspace directory name (bench.workspace_key) rather than the id, so the raw id stays raw everywhere it is data — the run id, the manifest row, the scoreboard, the --model flag. Two ids that sanitize to the same directory name still get different directories, because the key carries a hash of the raw id.

load_config also rejects any duplicate in models — two entries for one model would share every run id.

Then cfg = load_config(state / "config.yaml") — it validates the model list, fills in every threshold and repeat count, and raises ConfigError with the reason when something is wrong. Show the user the resolved Config, including which model will freeze the golden references.

Decide the target's mode now, because everything after this depends on it: git -C <target> rev-parse --is-inside-work-tree. A git target keeps working in place, on branches; a non-git target is copied once and worked on inside the copy.

Git target. Append .whetstone-mcp/ to the target's .gitignore and commit that change (or ask the user to), before anything else touches the repo:

cd <target> && git add .gitignore && git commit -m "chore: ignore .whetstone-mcp"

Step 6 calls require_clean_tree(repo) at every apply, and a modified or newly created .gitignore is an uncommitted change — leave it uncommitted and Gate 2 blocks on round 1 with ChangeError: dirty worktree. If the user does not want that commit on their branch, they must stash or commit it themselves before the first round; say so explicitly rather than leaving the tree dirty.

Non-git target. Do this now, before Step 2 introspects anything — every change tool in Step 6 is git, and running them against a non-repo dies at require_clean_tree with a raw git error. The whole optimization then runs against a copy, and the user's own directory is never modified:

cd <target>
rm -rf .whetstone-mcp/worktree
mkdir -p .whetstone-mcp/worktree
# everything except the harness's own state directory
rsync -a --exclude '.whetstone-mcp/' ./ .whetstone-mcp/worktree/
cd .whetstone-mcp/worktree
git init -q
git config user.name  "whetstone-mcp"        # local to this copy only
git config user.email "whetstone-mcp@localhost"
git add -A && git commit -q -m "whetstone-mcp: baseline copy of the target"
git rev-parse HEAD                            # keep this as <initial>

Then, for the rest of the run: server_cwd in config.yaml points at the matching directory inside the copy, and Step 6's repo is <target>/.whetstone-mcp/worktree. Set server_cwd before Step 2, or the baseline measures the original tree while the rounds change the copy. There is no .gitignore commit for such a target — .whetstone-mcp/ sits outside the copy, and the copy's own history starts at <initial>. Keep <initial>; Step 7 diffs against it to produce the patch.

Create the bench config directory now, and probe every model through the same environment a scored run gets — sanitized_env strips the API-key and alternate-endpoint variables (spec §5: subscription auth) and forces the CLAUDE_CODE_MAX_OUTPUT_TOKENS floor. A probe run under the raw shell environment can pass on an API key the scored runs will not have, or with a token ceiling they will not get, and then the phase fails for a reason the probe was supposed to catch:

import os, subprocess, time
from whetstone_mcp.bench import sanitized_env

config_dir = state / "bench" / "claude-config"
config_dir.mkdir(parents=True, exist_ok=True)
probe_env = sanitized_env(dict(os.environ), config_dir)   # sets CLAUDE_CONFIG_DIR

auth = subprocess.run(["claude", "auth", "status"], env=probe_env,
                      capture_output=True, text=True)
print(auth.returncode, auth.stdout, auth.stderr)

probe_seconds = {}
for model in cfg.models:
    started = time.monotonic()
    p = subprocess.run(["claude", "-p", "reply OK", "--model", model],
                       env=probe_env, capture_output=True, text=True)
    probe_seconds[model] = time.monotonic() - started
    if p.returncode != 0:
        raise SystemExit(f"model probe failed for {model!r}: {p.stderr.strip()}")

Every probe must succeed and claude auth status must report subscription auth. Keep probe_seconds; Step 5's cost estimate is built from it.

Seed the credentials first, on every platform (spec §5). The bench config dir is fresh, so claude auth status fails there even for a logged-in user. This includes macOS: the login is not inherited from the user's normal config dir, despite the shared Keychain. When the check fails, have the user run one interactive login with the bench config dir exported — the same CLAUDE_CONFIG_DIR as above:

CLAUDE_CONFIG_DIR=<target>/.whetstone-mcp/bench/claude-config claude login

Then re-run claude auth status in that directory and continue only once it reports subscription auth. The auth preflight is the check on every platform. Never copy credential files by hand.

claude login copies a plugin cache into the bench config dir. That cache is context-bearing, so audit_bench scrubs it — along with the projects/ transcript spool the CLI writes on every run — before each audit. Expect both to reappear and be removed again; that is the design, not a fault.

The full effective-toolset preflight comes after the bench exists, in Step 2.

Step 2 — Introspect and triage safety (done when every tool has a label)

introspect is async, so call it through asyncio.run:

import asyncio, os
from whetstone_mcp.introspect import introspect, IntrospectError
from whetstone_mcp.runner import RunSpec, run_phase

surface = asyncio.run(introspect(cfg.server_command, cfg.server_cwd,
                                 env=cfg.server_env, url=cfg.server_url))

That one call covers both target modes: with server_url set, server_command is empty and introspect connects over local HTTP instead of spawning anything. Pass all four arguments every time you introspect — here and again in Step 6 — so the two modes never diverge.

env=cfg.server_env is not optional. Without it the server is spawned with only the SDK's default inherited variables (HOME, LOGNAME, PATH, SHELL, TERM, USER), while the scored runs' server — spawned by the CLI from bench.mcp.json, which carries the same server_env — gets what the config declares. A server that reads any other variable would then behave one way when measured and another way when replayed, and Step 6's output-compat check would revert a good bundle for a reason no report could explain.

It performs the initialize handshake, records the server instructions and its declared capabilities, follows tools/list pagination, and calls nothing. Save surface to surface.json as asdict(surface) — it is the drafting input for Step 3, the schema baseline for Step 6's guardrails, and what a resumed session rebuilds the bench from:

from dataclasses import asdict
atomic_write_json(state / "surface.json", asdict(surface))

When the server exposes no tools, introspect raises IntrospectError. Report that the server has nothing to evaluate and stop here.

User gate 0. Present every tool — name, description, input schema, and any annotations — and ask the user to label each one:

• read_only — reads state, changes nothing.
• stateful_safe — mutates only local state that the Step 1 reset command restores. Config rejects this label when reset_command is null, because without one the definition is empty: the replay would see whatever state the previous call left behind, not the state the recorded output came from.
• destructive — external side effects.

Offer the tool's annotations as a suggested label and let the user confirm or correct it; annotations are a hint from the server author, not a trust boundary. Write the confirmed labels into config.yaml under tool_safety and re-run load_config.

When every tool is destructive, no execution is possible. Offer description-only mode: change classes 1–2 from references/polish-playbook.md, reviewed by the user, with no scoring loop — or stop.

Then build the bench:

bench = setup_bench(cfg, surface, state / "bench")   # default workspace root

It writes the bench settings file and bench.mcp.json and returns a BenchEnv. A tool reaches the bench allow list only through its tool_safety label, so a missing label silently removes that tool from every run — check the generated settings file against the labelled surface before moving on.

workspace_root is where every scored run actually executes, and it must be outside any git repository (spec §7.1, §11): a workspace under the target repo drags the target's CLAUDE.md — which likely describes the very tools under test — and its .claude/settings.local.json into every run, because local-settings resolution walks up to the git root. setup_bench refuses a root inside a repo with BenchError. Only the bench config dir and bench.mcp.json stay under <target>/.whetstone-mcp/bench.

Omit workspace_root and take the default. It is $XDG_CACHE_HOME/whetstone-mcp/workspaces/<hash of the bench root>, falling back to ~/.cache/..., and it is chosen for two properties the OS temp directory does not have. It is deterministic, so a resumed phase still resolves the absolute answer and transcript paths its manifest rows recorded; and it is durable, where macOS purges /var/folders on roughly a three-day cycle and TMPDIR changes between sessions. A temp-dir root strands an interrupted 540-session phase and breaks Step 6's re-parse of the baseline transcripts, which happens after every applied bundle. Pass an explicit root only when the user needs the runs somewhere specific, and then keep that same root for the whole optimization.

Preflight (spec §7.1, once per phase — and after every bench rebuild). Verify the effective toolset before spending a phase on it. Run one unscored probe session with the real command shape, in a throwaway workspace. At Gate 0, have the user approve one minimal schema-valid call to a non-destructive tool; use that exact bare name and arguments below. After a class-6 rebuild, probe a newly added safe tool so the new route itself is proven visible.

import hashlib, json
from whetstone_mcp.bench import audit_bench
from whetstone_mcp.pipeline import load_manifest_records, load_record_transcript
from whetstone_mcp.transcript import bare_tool_name

audit_bench(bench)   # no CLAUDE.md, skills, agents, project memory, or plugins
probe_tool = "<user-confirmed bare safe tool name>"
probe_args = {...}   # exact user-confirmed schema-valid arguments
spec = RunSpec("preflight", cfg.models[0], "preflight", 1,
               f'Call the MCP tool {probe_tool} exactly once with these exact '
               f'JSON arguments: {json.dumps(probe_args, sort_keys=True)}. '
               'Then write '
               '{"values": {"ok": true}, "assumptions": [], '
               '"needs_clarification": false} to answer.json.',
               allowed_tools=(probe_tool,))
preflight_key = hashlib.sha256(json.dumps({
    "allow": bench.allow, "deny": bench.deny, "target": bench.target,
}, sort_keys=True).encode()).hexdigest()[:12]
preflight_manifest = manifests / f"preflight-{preflight_key}.jsonl"
run_phase([spec], bench, cfg, preflight_manifest, dict(os.environ))
rec = load_manifest_records(preflight_manifest)[spec.run_id]
assert rec.status == "complete"
assert rec.answer_path.exists()      # the Write allow actually works
probe = load_record_transcript(rec)
unexpected = [c.name for c in probe.tool_calls
              if c.name != "Write" and not c.name.startswith("mcp__target__")]
matching = [c for c in probe.tool_calls
            if bare_tool_name(c.name) == probe_tool]
target_calls = [c for c in probe.tool_calls
                if c.name.startswith("mcp__target__")]
assert not unexpected
assert matching and all(not call.is_error for call in matching)
assert all(bare_tool_name(call.name) == probe_tool for call in target_calls)

Do not treat the model's prose list as evidence of visibility: allowedTools is permission, not discovery. The authoritative preflight evidence is audit_bench's exact generated rule/config inspection, the real command's --strict-mcp-config, the programmatically parsed required MCP call, and a successful answer write. No builtin call other than the single scoped Write may occur, no destructive/unlabelled target tool may occur, and answer.json must exist. The write assertion is not optional — a permission rule that silently blocks the answer path turns every run in a 540-session phase into no_answer, and the harness then reports instant success having measured nothing. If the file is missing, stop and fix the rules; do not start the phase. Record the probe summary and transcript, do not score it. Both run_phase and baseline call audit_bench again, so every resumed phase rechecks isolation instead of relying on the Step 2 snapshot.

Step 3 — Draft the task set (done when the user approves the list at gate 1)

Read references/task-authoring.md first: it holds the task schema, the anti-leakage rules, and the stratified 40 scored / 20 held-out split.

Draft the set with a strong-model claude -p call over surface.json, following that document. Scored tasks and held-out tasks go in different directories — that separation is the held-out hygiene mechanism (spec §8), not a filing preference:

from whetstone_mcp.tasks import dump_task

tasks_dir   = state / "tasks"              # the 40 scored tasks
holdout_dir = tasks_dir / "holdout"        # the 20 held-out tasks
for t in drafted:                          # the Task objects the drafting call produced
    dump_task(t, holdout_dir if t.holdout else tasks_dir)

tasks = load_task_set(tasks_dir, holdout_dir,
                      cfg.scored_count, cfg.holdout_count)
scored_tasks = [t for t in tasks if not t.holdout]
holdout_tasks = [t for t in tasks if t.holdout]

load_task_set uses the same non-recursive loads, then also rejects duplicate ids across partitions, wrong holdout flags, and count drift. A Step 6 failure analysis still receives only scored_tasks; code that legitimately needs all 60 uses tasks. Keep these three names (tasks_dir, holdout_dir, tasks) for the whole optimization; later steps use them.

Then run the anti-leakage review. Dispatch a fresh subagent that has not seen your drafting reasoning, and give it exactly three things: the tool surface, the drafted tasks, and the anti-leakage rules quoted in references/task-authoring.md. Ask it to name every task whose prompt breaks a rule and to say which rule. Rewrite the flagged prompts and review again until the reviewer returns nothing.

User gate 1. Present the full list — id, tier, prompt, expected values, route, held-out flag — and ask for approval or edits. Apply the edits, then send the edited tasks back through the anti-leakage review. Any task the user edits after Step 4 returns here and is verified again.

Step 4 — Verify and freeze the references (done when every kept task is stable in references.json)

Pick the reference effort level first (spec §6.3: reference runs execute "at maximum effort"). The installed CLI takes --effort <level>, and RunSpec.effort is what puts it on the command line. Read the accepted levels from the CLI's own help text — never by probing the API:

claude --help | grep -i -A2 -- "--effort"

Set REF_EFFORT to the highest level that text lists. If it names no levels, use "max" when the word appears in the help text and "high" otherwise. Record the level you used in this run's first report, and note it under "Effort level for reference runs" in references/model-traps.md when the level you observe differs from the one recorded there. Scored runs never get an effort level — RunSpec.effort stays None everywhere except here, so the baseline measures each model the way the user would invoke it.

For each approved task, build the configured reference repeats and execute them. A task marked human_verified is still executed: its values are never re-frozen, but a later prompt/route edit must still prove that the task is executable and stable.

import asyncio, json, os
from whetstone_mcp.guardrails import (
    CorpusEntry, merge_corpus, record_corpus, save_corpus,
)
from whetstone_mcp.freeze import (extract_reference_values, freeze_reference,
                                  write_manifest)
from whetstone_mcp.pipeline import (load_manifest_records,
                                    load_record_transcript)
from whetstone_mcp.runner import RunSpec, run_phase
from whetstone_mcp.tasks import dump_task

REF_EFFORT = "high"          # the level you just read from `claude --help`
corpus_entries: list[CorpusEntry] = []
normalization_rules: dict[str, list[str]] = {}
workflow = checkpoint(state_path, workflow, workflow.stage,
                      ref_effort=REF_EFFORT)

specs = [RunSpec(run_id=f"ref_{t.id}_{r}", model=cfg.reference_model,
                 task_id=t.id, repeat=r, prompt=t.prompt, effort=REF_EFFORT)
         for t in tasks
         for r in range(1, cfg.ref_repeats + 1)]
run_phase(specs, bench, cfg, manifests / "reference.jsonl", dict(os.environ))

run_phase writes a manifest row per run, so an interrupted phase resumes by calling it again with the same manifest path. Wrap this call in the RateLimitPause handler above; a paused phase resumes the same way.

Give every phase its own manifest path, and never reuse one for a different phase. The manifest file name is the phase identity: run_phase derives the workspace subdirectory from it, and run ids repeat across phases by construction ({model}_{task}_{repeat}). Distinct names are what keep a round's re-run from clearing the baseline's workspaces — Step 6 re-parses every baseline transcript after each applied bundle, and those paths must still resolve in round 5. A resumed phase reuses its own name, which is how resume works at all.

run_phase returns only the records it executed on this call, so never freeze from its return value: on a resumed phase that is a subset, and the task would be frozen from two runs instead of three without saying so. Read every row back from the manifest and freeze in one loop over tasks:

records = load_manifest_records(manifests / "reference.jsonl")   # {run_id: RunRecord}
freeze_results = []          # one FreezeResult per task, in `tasks` order
missing_runs = []            # planned run ids the phase never recorded

for task in tasks:
    run_values = []
    for r in range(1, cfg.ref_repeats + 1):
        rec = records.get(f"ref_{task.id}_{r}")
        if rec is None:
            missing_runs.append(f"ref_{task.id}_{r}")
            run_values.append(None)
        elif (rec.status != "complete" or rec.timed_out or rec.exit_code != 0
              or not rec.answer_path.exists()):
            # C2: a killed or non-zero-exit session is not a reference, even
            # when it left an answer.json behind. Freezing from one would make
            # the golden value the output of a run the harness cut short.
            run_values.append(None)      # freeze_reference -> "failed"
        else:
            # Full contract, answer semantics, and required/forbidden route
            # are checked before a value can enter the golden freeze.
            run_values.append(extract_reference_values(
                task, rec.answer_path, load_record_transcript(rec)))
    freeze_results.append(freeze_reference(task, run_values))

by_result = {fr.task_id: fr for fr in freeze_results}

If missing_runs is non-empty the phase is incomplete, not failed: those runs never recorded a row at all. Re-run the run_phase call above (it resumes from the manifest) before freezing anything, rather than reading a missing id as a run that produced no answer.

Then act on each task's result = by_result[task.id]:

• status == "stable" — keep the task and replace its expected.values with result.frozen_values via dump_task(task, holdout_dir if task.holdout else tasks_dir). Every write-back in this skill uses that same expression: writing a held-out task into tasks_dir would put it in front of the Step 6 failure analysis.
• status is "unstable" or "failed" — rewrite the task's prompt to remove the ambiguity, or drop it. Return to Step 3 for a replacement so the 40/20 split and the tier mix stay intact. "unstable" also covers a task the three runs agreed on but whose answer is not expressible in the spec §6.4 types (a nested object, a list of numbers): writing that back would produce a task file load_tasks can no longer read, so the task is rewritten to ask for something checkable, or dropped.
• result.drafted_disagrees is true — show the user the drafted value beside the frozen one and let them choose. A value the user fixes by hand gets human_verified: true and stays exactly as the user set it; write that flag into the task file, and every later verification pass will preserve it instead of re-freezing over it.

Write the manifest of everything stable (it skips non-stable results itself):

write_manifest(freeze_results, state / "references.json")

Record the guardrail corpus from these same transcripts. Load each with load_record_transcript (the complete normalized summary, not the capped diagnostic JSONL), and for every non-destructive tool pick one ToolCall whose is_error is false and whose result text is not an in-band error. The transcript supplies the reviewed input only. Pass every selected input through record_corpus once so text, structuredContent, and isError are captured from the MCP result without flattening. Keep the fingerprint of each manually identified in-band failure for Step 6's class-3 check.

When a non-destructive tool never appears in any transcript, it has no corpus entry, so nothing would catch a schema tightening or an output change on it for the rest of the optimization. Cover it explicitly: draft one minimal valid call from the tool's input schema (required fields only, values the schema plainly admits), and show it to the user in the corpus summary that closes this step, alongside the tools covered from transcripts — the same kind of review Gate 1 gave the task list. Ask them to confirm the call is safe and sensible for this server before it is ever made. Then record its output by recording it — through record_corpus, never by calling the tool by hand:

draft = [CorpusEntry(tool="<name>", args={...}, output_text=None,
                     no_replay=False)]
recorded = asyncio.run(record_corpus(
    draft, cfg.tool_safety, cfg.server_command, cfg.server_cwd,
    reset_command=cfg.reset_command, server_env=cfg.server_env,
    server_url=cfg.server_url, reset_timeout_s=cfg.reset_timeout_s))
corpus_entries = merge_corpus(corpus_entries, recorded)

record_corpus and replay_corpus share the same destructive/unlabelled barrier (spec §6.2) — a tool the user has not labelled read_only or stateful_safe raises GuardrailError before any session opens, as does a stateful_safe tool with no reset_command. no_replay means "record once, do not call again during future guardrails"; it does not bypass the initial user-approved recording. If the user will not approve even one fixture-data call, classify that tool destructive so neither scored runs nor the harness can execute it.

Decide each tool's normalization rules at the same time, by comparing its result text across the three reference runs. The three legal rule strings are "timestamps", "uuids" and "durations"; list the ones that account for the differences you see, use [] when the text is identical every time, and when something else varies mark the tool no_replay instead — a no_replay tool needs no rules. Always pass no_replay=False to record_corpus; after it returns the complete recorded text/structured/error triple, set recorded_entry.no_replay = True for an inherently unstable tool before save_corpus.

Write all of it to .whetstone-mcp/corpus.json as a list of objects with exactly these seven keys, which is the file Step 6 reads. Preserve text, structuredContent, and protocol error status independently:

[
  {"tool": "lookup", "args": {"key": "alpha"}, "output_text": "1.5",
   "output_structured": null, "is_error": false,
   "no_replay": false, "normalize_rules": []},
  {"tool": "status", "args": {}, "output_text": "ready since 2026-08-30T09:00:00Z",
   "output_structured": {"ready": true}, "is_error": false,
   "no_replay": false, "normalize_rules": ["timestamps"]},
  {"tool": "sample", "args": {}, "output_text": "0.7314",
   "output_structured": null, "is_error": false,
   "no_replay": true, "normalize_rules": []}
]

After every tool has exactly one entry and every normalization decision has been put in normalization_rules, use the validated atomic writer (never a hand-built partial JSON append):

safe_tools = {
    tool.name for tool in surface.tools
    if cfg.tool_safety.get(tool.name) in ("read_only", "stateful_safe")
}
if {entry.tool for entry in corpus_entries} != safe_tools:
    raise RuntimeError(
        "corpus must contain exactly one entry for every non-destructive tool")
if set(normalization_rules) != safe_tools:
    raise RuntimeError(
        "record an explicit normalization-rule list (including []) for every "
        "non-destructive tool")
save_corpus(state / "corpus.json", corpus_entries, normalization_rules)

Step 5 — Cost gate and baseline (done when the manifest covers every planned run)

Cost gate. Show the user the run count — len(cfg.models) × <task count> × cfg.repeats sessions — and a wall-clock estimate from the Step 1 probe times. Wait for an explicit go.

from whetstone_mcp.pipeline import baseline
from whetstone_mcp.scoring import gap_tasks, roles
results = baseline(cfg, tasks, bench, manifests / "baseline.jsonl",
                   dict(os.environ))          # -> list[ModelResult]
target, ceiling = roles(results, cfg.model_cost_order)
gap = gap_tasks(results)

baseline builds the run specs, drives run_phase, and scores every complete record. Keep results under the name before — it is the "before" column of the final report and the baseline every round's deltas are measured against — and write the whole starting state to disk before doing anything else with it:

before = results          # the Step 5 baseline, kept for Steps 6 and 7
latest = before

# The initial state.json: round 0, nothing applied, nothing set aside, and
# the plateau history seeded with the baseline's target count. Step 6 and
# Step 7 read every one of their names back out of this file — including on
# the very first round of a session that never stopped — so this write is
# what makes the rest of the optimization runnable at all.
target_result = next(r for r in latest if r.model == target)
workflow = checkpoint(
    state_path, workflow, workflow.stage,
    before=model_results_to_rows(before),
    latest=model_results_to_rows(latest),
    applied=[], remaining_findings=[],
    plateau=[[0, target, target_result.holdout_passes]],
    round_n=0, branch="",
    report_manifests=[str(manifests / "baseline.jsonl")])

Step 4 already wrote ref_effort. Reload WorkflowState once and assert that before, latest, and ref_effort are populated before moving on.

Check the stop rule before starting round 1. A server can already be good enough for the target model, and a round spent on a scoreboard that already passes costs sessions and changes a repository for nothing:

from whetstone_mcp.scoring import stop_success
if stop_success(before, cfg.gap_threshold_tasks, cfg.floor_fraction,
                cfg.model_cost_order):
    ...   # skip Step 6 entirely and go to Step 7

When it is already true, say so plainly — the gap and the floor, with the numbers — and go straight to Step 7. The final report's before and after columns will be two independent measurements of the unchanged server, which is an honest and useful thing to hand the user; do not manufacture a round to fill the report out.

Wrap the call in the RateLimitPause handler above: a 540-session baseline is the phase most likely to hit the window, and re-running this step later resumes from baseline.jsonl.

For any run you want to inspect on its own, score its manifest record with score_record(task, rec) from whetstone_mcp.pipeline, and roll a model's per-task scores up with aggregate(model, per_task, holdout_ids). per_task maps a task id to a list of (repeat index, RunScore) pairs — the repeat as planned (1-based), not the position in the list. Only complete runs are scored, so labelling the repeat is what keeps a task missing repeat 2 from reporting repeat 3's run in repeat 2's column:

from whetstone_mcp.pipeline import load_manifest_records, score_record
records = load_manifest_records(manifests / "baseline.jsonl")
per_task = {}
for t in tasks:
    runs = []
    for r in range(1, cfg.repeats + 1):
        rec = records.get(f"{model}_{t.id}_{r}")        # `model` is the id you are inspecting
        if rec is not None and rec.status == "complete":
            runs.append((r, score_record(t, rec)))
    per_task[t.id] = runs                               # every planned task, even an empty one

score_record is score_run plus the record's own verdict (C2): a run the harness killed at the timeout scores check_failed / timeout_with_output even when a matching answer.json is sitting in the workspace. A non-timeout non-zero CLI exit is an infra_failed record and is retried, regardless of stdout. baseline never scores such a row. For a record-less inspection use score_run(task, answer_path, load_record_transcript(rec)); never score the capped diagnostic JSONL when a complete summary exists.

aggregate computes spread over the held-out tasks in holdout_ids only, whatever else the phase ran, so the column means the same thing in every report (I4). A phase with no held-out task returns an empty spread.

Step 6 — Round loop (done when a report, Gate 2 outcome, and durable stage exist)

Run the reload block first, then dispatch on workflow.stage. Do not execute a section whose stage has already passed. This is the executable resume rule, not merely record keeping.

A. Create a new report (idle or complete only)

Increment the round only here. Build the scoreboard in configured strength order and break a target tie with the explicit cheapest-first order:

from whetstone_mcp.pipeline import manifest_infra_reruns
from whetstone_mcp.report import append_gate_outcome, render_round_report
from whetstone_mcp.scoring import gap_tasks, roles

round_n = workflow.round_n + 1
results = model_results_from_rows(workflow.latest)
target, ceiling = roles(results, cfg.model_cost_order)
target_before = next(r.holdout_passes for r in results if r.model == target)

Analyze only scored-task failures. Use load_record_transcript(rec), so a large stream is read from its complete summary rather than its capped log. Inspect routes, inputs, protocol errors, explicit in-band error text, scoring subcodes, and output tokens; group failures by shared server-side cause. Record the fingerprints (guardrails.call_fingerprint) of reviewed in-band failures so a class-3 compatibility check does not mistake them for successful calls.

Now run the proposal tournament described in references/polish-playbook.md, then judge it. Three proposers draft against the same failure clusters with fixed biases — cheap-classes, workflow, and regression-hunter, the last reading the first two — and you dedupe by failure cluster, break ties toward the cheaper class, and keep what survives. Skip the tournament and draft once when the round has fewer than three distinct clusters; say so in the report rather than staging a tournament over one obvious fix.

Read the calibration ledger before you rank, and discount every predicted_delta by the bias it reports:

from whetstone_mcp.calibration import render as render_calibration, summarize
bias = summarize(calibration)["mean_error"]     # None until two scored rounds

A positive mean_error means past rounds over-promised by that many held-out passes; subtract it before ranking. Below two scored rounds there is no bias to apply, and summarize says so by returning None — rank on the raw predictions and do not invent a correction.

Draft ranked Proposal objects per references/polish-playbook.md. Every proposal must contain:

• one exact unified diff;
• base_commit equal to the current full git rev-parse HEAD;
• a sha256 for exactly every touched existing path, or "new" for a created path;
• any task_bumps, plus an exact bump_routes mapping from every bumped task id to the newly admitted bare-tool group.

All proposals in one report use the same base commit. Combine proposals that touch the same path: application order is not something the user approved. For every proposed new explicit default, inspect baseline calls to that same tool where the argument was omitted. The default must equal the behavior those calls observed. Put that evidence in the proposal rationale; when no omitted- argument call exists, flag the default for explicit user judgment at Gate 2 instead of claiming it is regression-backed (spec §10.4's review step). Render and checkpoint the report:

report_path = state / "reports" / f"round-{round_n}.md"
pending_bumps = sorted({tid for p in proposals for tid in p.task_bumps})
holdout_ids = {task.id for task in holdout_tasks}
unknown_bumps = set(pending_bumps) - set(by_id)
if unknown_bumps:
    raise RuntimeError(f"task bumps name unknown ids: {sorted(unknown_bumps)}")
if set(pending_bumps) & holdout_ids:
    raise RuntimeError(
        "adaptive proposals may not name or version-bump held-out tasks; "
        "keep the headline ruler fixed and bump scored tasks only")
ruler_note = ("version/route bump proposed for " + ", ".join(pending_bumps)
              if pending_bumps else None)
report_text = render_round_report(
    round_n, results, findings, proposals,
    manifest_infra_reruns([Path(p) for p in workflow.report_manifests]),
    ruler_note, baseline=before)
report_text += ("\n\n## Judge calibration\n\n"
                + render_calibration(calibration) + "\n")
tmp = report_path.with_suffix(".md.tmp")
tmp.write_text(report_text)
os.replace(tmp, report_path)
workflow = checkpoint(
    state_path, workflow, "report_ready", round_n=round_n,
    target=target, ceiling=ceiling, report_path=str(report_path),
    ruler_note=ruler_note or "",
    shown_proposals=proposals_to_rows(proposals),
    approved_proposals=[], shas={}, bump_routes={}, task_backups={},
    reverify_manifest="", holdout_manifest="", scored_manifest="",
    in_band_failure_fingerprints=sorted(in_band_failures),
    rejection_status="", rejection_violations=[],
    restore_rearm_artifacts=False, source_reverted=False,
    runtime_restore_confirmed=False)

Only held-out counts may appear; never held-out task ids or transcripts. This is why adaptive task-version bumps are limited to scored tasks. A workflow tool that cannot satisfy a held-out task's already-frozen route is evaluated as a new post-optimization eval version, not slipped into the headline ruler. Infrastructure re-runs are total invocations beyond the first planned invocation, including a later resume of an infra_failed row.

B. Gate 2 (report_ready)

Show the saved report and ask for approve all, selected, none, or stop. Explain that attribution is to the selected bundle. If there is no target test command, classes 3–6 also require explicit acceptance of that regression risk.

For a selection, recover the exact proposal objects already shown—do not regenerate them—and checkpoint before touching git. Reject an unknown id and preserve report order:

shown = proposals_from_rows(workflow.shown_proposals)
selected_ids = {...}  # the exact ids the user approved
if not selected_ids <= {proposal.id for proposal in shown}:
    raise RuntimeError("approval named an unknown proposal")
approved = [proposal for proposal in shown if proposal.id in selected_ids]
if not approved:
    raise RuntimeError("an empty selection is 'approved none', not a bundle")
workflow = checkpoint(
    state_path, workflow, "approved",
    approved_proposals=proposals_to_rows(approved), gate_outcome="approved")

If the user approves none or stops, call append_gate_outcome(Path(workflow.report_path), "approved none", [], []), checkpoint rejected and then complete, and stop. Do not open a branch, build, replay guardrails, bump tasks, or rerun unchanged models.

append_gate_outcome(Path(workflow.report_path), "approved none", [])
workflow = checkpoint(state_path, workflow, "rejected",
                      gate_outcome="approved none")
workflow = checkpoint(state_path, workflow, "complete")

C. Apply the exact bundle (approved / branch_opened)

repo is the git target or Step 1's git worktree copy. Branch creation is idempotent and separately checkpointed:

from whetstone_mcp.changes import (
    apply_bundle, BundleRejected, ensure_round_branch, require_clean_tree,
    revert_bundle,
)
from whetstone_mcp.guardrails import Violation

approved = proposals_from_rows(workflow.approved_proposals)
base_commit = approved[0].base_commit
if workflow.stage == "approved":
    require_clean_tree(repo)
    branch = ensure_round_branch(repo, workflow.round_n, base_commit)
    workflow = checkpoint(state_path, workflow, "branch_opened", branch=branch)
if workflow.stage == "branch_opened":
    try:
        shas = apply_bundle(repo, approved)
    except BundleRejected as error:
        append_gate_outcome(
            Path(workflow.report_path), "apply failed; bundle rejected",
            approved, [Violation("apply", "", str(error))])
        workflow = checkpoint(
            state_path, workflow, "rejected",
            gate_outcome="apply failed; bundle rejected")
        workflow = checkpoint(state_path, workflow, "complete")
        raise SystemExit(0)
    else:
        workflow = checkpoint(state_path, workflow, "applied", shas=shas)

apply_bundle rejects an unrelated changed HEAD, missing/extra base hashes, overlapping proposal paths, staged content, filter-rewritten blobs, and any committed path or byte not represented by the approved diff. If the process stopped after a proposal commit but before the applied checkpoint, it recomputes the exact approved Git tree in a temporary index, accepts only that linear proposal prefix, and resumes the rest without duplicating commits. When that prefix is the whole bundle there is nothing left to apply, so the resume returns the recovered commit map without requiring a clean tree. A missing base commit is always an error.

D. Build and guardrails (applied)

Build first with the bounded process-group runner. For a URL target, have the user confirm that the rebuilt local process was restarted before continuing. Re-introspect and load the seven-key corpus strictly:

from whetstone_mcp.guardrails import (
    CorpusEntry, Violation, check_baseline_args, check_corpus_acceptance,
    load_corpus, replay_corpus, run_test_suite,
)

built, build_log = (run_test_suite(cfg.build_command, cfg.server_cwd,
                                   cfg.run_timeout_s)
                    if cfg.build_command else (True, ""))
violations = [] if built else [Violation("build", "", build_log)]
if built:
    new_surface = asyncio.run(introspect(
        cfg.server_command, cfg.server_cwd, cfg.server_env, cfg.server_url))
    new_schemas = {tool.name: tool.input_schema for tool in new_surface.tools}
else:
    new_surface, new_schemas = None, {}

corpus, rules = load_corpus(state / "corpus.json")
if built:
    violations += check_corpus_acceptance(corpus, new_schemas)

For class 3, pool calls from every complete baseline row. Protocol errors are excluded in code; pass explicitly reviewed fingerprints to exclude in-band failures without guessing from generic words:

baseline_records = load_manifest_records(manifests / "baseline.jsonl")
historical_calls = [
    call
    for rec in baseline_records.values() if rec.status == "complete"
    for call in load_record_transcript(rec).tool_calls
]
if built and any(p.change_class == 3 for p in approved):
    violations += check_baseline_args(
        historical_calls, new_schemas,
        set(workflow.in_band_failure_fingerprints))

if built:
    violations += asyncio.run(replay_corpus(
        corpus, cfg.tool_safety, cfg.server_command, cfg.server_cwd, rules,
        reset_command=cfg.reset_command, server_env=cfg.server_env,
        server_url=cfg.server_url, reset_timeout_s=cfg.reset_timeout_s,
        reset_env=dict(os.environ)))
if built and cfg.test_command:
    tests_ok, test_log = run_test_suite(
        cfg.test_command, cfg.server_cwd, cfg.run_timeout_s)
    if not tests_ok:
        violations.append(Violation("test_suite", "", test_log))

Replay compares text, structured content, and isError independently under the additive-only rule. It runs resets with the configured environment and a timeout. Persist the rejection evidence before touching Git; the shared rollback dispatcher below then reverts every selected commit newest-first. There is no rerun after a revert and no invented per-proposal attribution. The manual explicit-default review described in section A is part of this guardrail result: an unverified default without the Gate-2 judgment it asked for is a Violation("default_review", <tool>, <detail>) and rejects the bundle; do not silently omit that check because it is agent-reviewed rather than a separate harness function.

if violations:
    workflow = checkpoint(
        state_path, workflow, "revert_pending",
        rejection_status="rejected and reverted",
        rejection_violations=[
            {"check": v.check, "tool": v.tool, "detail": v.detail}
            for v in violations
        ],
        restore_rearm_artifacts=False, source_reverted=False,
        runtime_restore_confirmed=not bool(cfg.server_url))
else:
    workflow = checkpoint(state_path, workflow, "guardrails_passed")

E. Re-arm and prove ruler changes (guardrails_passed onward)

Re-introspect (a new session may not have D's local variable), persist the surface with asdict, classify every added tool with the user, reload config, rebuild the bench, call audit_bench, and repeat the Step 2 preflight. Report removed tools and changed annotations. For every new non-destructive tool, have the user approve one minimal schema-valid call and use record_corpus to write a full seven-key entry. Merge corpus entries by tool name and replace the file atomically, so resuming this stage cannot append duplicates. Destructive tools remain denied and cannot become a scored route. Then checkpoint rearmed:

from dataclasses import asdict
from whetstone_mcp.guardrails import (
    load_corpus, merge_corpus, record_corpus, run_test_suite, save_corpus,
)

if workflow.stage == "guardrails_passed":
    old_surface = surface
    surface = asyncio.run(introspect(
        cfg.server_command, cfg.server_cwd, cfg.server_env, cfg.server_url))
    old_tools = {tool.name: tool for tool in old_surface.tools}
    new_tools = {tool.name: tool for tool in surface.tools}
    added_names = sorted(set(new_tools) - set(old_tools))
    removed_names = sorted(set(old_tools) - set(new_tools))
    changed_annotations = sorted(
        name for name in set(old_tools) & set(new_tools)
        if old_tools[name].annotations != new_tools[name].annotations)
    print("added", added_names, "removed", removed_names,
          "annotation changes", changed_annotations)
    atomic_write_json(state / "surface.json", asdict(surface))
    # Show additions and changed annotations, collect the user's safety labels,
    # and re-run load_config before allowing any new call. For each approved
    # non-destructive addition, create one minimal CorpusEntry in `drafts`
    # with no_replay=False; only mark a returned recorded entry no_replay.
    drafts: list[CorpusEntry] = []
    addition_rules: dict[str, list[str]] = {}
    cfg = load_config(state / "config.yaml")
    safe_additions = {
        name for name in added_names
        if cfg.tool_safety.get(name) in ("read_only", "stateful_safe")
    }
    if {entry.tool for entry in drafts} != safe_additions:
        raise RuntimeError(
            "drafts must contain exactly one approved corpus call for every "
            "new non-destructive tool")
    corpus, rules = load_corpus(state / "corpus.json")
    additions = asyncio.run(record_corpus(
        drafts, cfg.tool_safety, cfg.server_command, cfg.server_cwd,
        reset_command=cfg.reset_command, server_env=cfg.server_env,
        server_url=cfg.server_url, reset_timeout_s=cfg.reset_timeout_s,
        reset_env=dict(os.environ)))
    corpus = merge_corpus(corpus, additions)
    rules.update(addition_rules)
    save_corpus(state / "corpus.json", corpus, rules)
    bench = setup_bench(cfg, surface, state / "bench")
    audit_bench(bench)
    # repeat the Step 2 probe here
    workflow = checkpoint(state_path, workflow, "rearmed")

Consolidate the approved bump_routes by task id; conflicting groups for one task are an error. Verify every named tool exists, is read_only or stateful_safe, and is in bench.allow. Before editing, save the exact YAML in task_backups; then increment each affected task once and append its approved route group:

new_schemas = {tool.name: tool.input_schema for tool in surface.tools}
if workflow.stage == "rearmed":
    bump_routes = {}
    for proposal in approved:
        for tid, group in proposal.bump_routes.items():
            if tid in bump_routes and bump_routes[tid] != group:
                raise RuntimeError(f"conflicting approved routes for {tid}")
            bump_routes[tid] = group
    if bump_routes:
        task_backups = {}
        for tid, group in bump_routes.items():
            task = by_id[tid]
            path = (holdout_dir if task.holdout else tasks_dir) / f"{tid}.yaml"
            task_backups[tid] = path.read_text()
            for name in group:
                assert name in new_schemas
                assert cfg.tool_safety.get(name) in ("read_only", "stateful_safe")
                assert f"mcp__target__{name}" in bench.allow
        workflow = checkpoint(
            state_path, workflow, "bump_pending", bump_routes=bump_routes,
            task_backups=task_backups)

if workflow.stage == "bump_pending":
    # Make the edit idempotent across a crash anywhere in this loop: restore
    # the pre-edit snapshot first, then derive every edit from it.
    for tid, text in workflow.task_backups.items():
        task = by_id[tid]
        path = (holdout_dir if task.holdout else tasks_dir) / f"{tid}.yaml"
        tmp = path.with_suffix(".yaml.tmp")
        tmp.write_text(text)
        tmp.replace(path)
    tasks = load_task_set(tasks_dir, holdout_dir,
                          cfg.scored_count, cfg.holdout_count)
    by_id = {task.id: task for task in tasks}
    for tid, group in workflow.bump_routes.items():
        task = by_id[tid]
        task.version += 1
        if group not in task.route.required_any:
            task.route.required_any.append(group)
        dump_task(task, holdout_dir if task.holdout else tasks_dir)
    workflow = checkpoint(state_path, workflow, "tasks_bumped")

Every bumped task is reference-verified through the new route, not merely re-executed while an old alternative remains available. Save the unique manifest first. RunSpec.allowed_tools denies every old safe tool for this run, and extraction also requires the exact group to appear:

from whetstone_mcp.freeze import (
    extract_reference_values, freeze_reference, update_manifest,
)

if workflow.bump_routes and workflow.stage == "tasks_bumped":
    reverify = manifests / f"round-{workflow.round_n}-reverify.jsonl"
    workflow = checkpoint(state_path, workflow, "reverify_pending",
                          reverify_manifest=str(reverify))
if workflow.stage == "reverify_pending":
    specs = [
        RunSpec(f"round{workflow.round_n}_ref_{tid}_{repeat}",
                cfg.reference_model, tid, repeat, by_id[tid].prompt,
                REF_EFFORT, tuple(group))
        for tid, group in workflow.bump_routes.items()
        for repeat in range(1, cfg.ref_repeats + 1)
    ]
    run_phase(specs, bench, cfg, Path(workflow.reverify_manifest),
              dict(os.environ))
    records = load_manifest_records(Path(workflow.reverify_manifest))
    verified = []
    for tid, group in workflow.bump_routes.items():
        task = by_id[tid]
        values = []
        for repeat in range(1, cfg.ref_repeats + 1):
            rec = records.get(
                f"round{workflow.round_n}_ref_{tid}_{repeat}")
            good = (rec is not None and rec.status == "complete"
                    and not rec.timed_out and rec.exit_code == 0
                    and rec.answer_path.exists())
            values.append(
                extract_reference_values(
                    task, rec.answer_path, load_record_transcript(rec),
                    set(group))
                if good else None)
        verified.append(freeze_reference(task, values))

If any result is not stable or disagrees with the existing frozen value, checkpoint the rollback before changing Git. The shared dispatcher atomically restores every task backup, reverts the whole server bundle, appends a rejection outcome, and stops. A route/version bump does not authorize changing the expected values. For stable, agreeing results, preserve the existing frozen values and call update_manifest(verified, state / "references.json"). It merges only the reverified ids instead of deleting every other golden reference.

if any(result.status != "stable" or result.drafted_disagrees
       for result in verified):
    workflow = checkpoint(
        state_path, workflow, "revert_pending",
        rejection_status="route verification failed; reverted",
        rejection_violations=[{
            "check": "route_verification", "tool": "",
            "detail": "new route did not stably reproduce the frozen value",
        }],
        restore_rearm_artifacts=True, source_reverted=False,
        runtime_restore_confirmed=not bool(cfg.server_url))
else:
    for result in verified:
        # Verification proves the new route returns the already-approved
        # ruler; it never silently moves that ruler to a new median/mode.
        result.frozen_values = dict(by_id[result.task_id].expected.values)
    update_manifest(verified, state / "references.json")

Shared rollback dispatcher (revert_pending)

This block runs before section F whenever either guardrails or forced-route verification checkpointed a rejection. revert_bundle is idempotent: it recognizes already-created revert commits and aborts/retries only an interrupted revert belonging to this exact bundle. Task restoration, rebuild, surface/corpus repair, and report append are likewise safe to repeat.

For a URL target, stop after source_reverted becomes true and ask the user to rebuild/restart the reverted local server. Only after explicit confirmation checkpoint runtime_restore_confirmed=True and resume this block.

# Run this only in the turn after that explicit URL-server confirmation.
workflow = checkpoint(state_path, workflow, "revert_pending",
                      runtime_restore_confirmed=True)

from whetstone_mcp.guardrails import Violation

if workflow.stage == "revert_pending":
    for tid, text in workflow.task_backups.items():
        task = by_id[tid]
        path = (holdout_dir if task.holdout else tasks_dir) / f"{tid}.yaml"
        tmp = path.with_suffix(".yaml.tmp")
        tmp.write_text(text)
        os.replace(tmp, path)

    if not workflow.source_reverted:
        revert_bundle(repo, [workflow.shas[p.id] for p in approved])
        workflow = checkpoint(state_path, workflow, "revert_pending",
                              source_reverted=True)

    if cfg.server_url and not workflow.runtime_restore_confirmed:
        raise SystemExit(
            "source reverted; rebuild/restart the local URL server, confirm "
            "that action, then resume rollback")

    if cfg.build_command:
        restored, restore_log = run_test_suite(
            cfg.build_command, cfg.server_cwd, cfg.run_timeout_s)
        if not restored:
            raise RuntimeError(
                "source is reverted but rebuilding its runtime failed; "
                f"fix the build and resume rollback: {restore_log}")

    surface = asyncio.run(introspect(
        cfg.server_command, cfg.server_cwd, cfg.server_env, cfg.server_url))
    atomic_write_json(state / "surface.json", asdict(surface))
    live_names = {tool.name for tool in surface.tools}
    corpus, rules = load_corpus(state / "corpus.json")
    corpus = [entry for entry in corpus if entry.tool in live_names]
    rules = {name: value for name, value in rules.items()
             if name in live_names}
    save_corpus(state / "corpus.json", corpus, rules)
    bench = setup_bench(cfg, surface, state / "bench")
    audit_bench(bench)

    saved_violations = [
        Violation(**row) for row in workflow.rejection_violations
    ]
    append_gate_outcome(Path(workflow.report_path),
                        workflow.rejection_status, approved,
                        saved_violations)
    workflow = checkpoint(
        state_path, workflow, "rejected",
        gate_outcome=workflow.rejection_status)
    workflow = checkpoint(state_path, workflow, "complete")
    raise SystemExit(0)

F. Rerun and close (rearmed or successful reverify_pending)

Before sessions, save both phase paths under rerun_pending. When there were bumps, write this transition only after stable verification and its manifest update. On resume, reuse the paths and do not rebuild, reapply, rebump, or reverify:

if workflow.stage in {"rearmed", "reverify_pending"}:
    holdout_manifest = manifests / f"round-{workflow.round_n}-holdout.jsonl"
    scored_manifest = (manifests / f"round-{workflow.round_n}-scored.jsonl"
                       if workflow.round_n % 2 == 0 else None)
    workflow = checkpoint(
        state_path, workflow, "rerun_pending",
        holdout_manifest=str(holdout_manifest),
        scored_manifest=str(scored_manifest) if scored_manifest else "")

Run the round-entry target and ceiling. If they are the same after a tie, include one other configured model as a harmless companion because the validated internal config requires two distinct ids:

run_models = list(dict.fromkeys([workflow.target, workflow.ceiling]))
if len(run_models) == 1:
    run_models.append(next(m for m in cfg.models if m != run_models[0]))
sub = Config.model_validate({**cfg.model_dump(),
    "models": run_models,
    "model_cost_order": [m for m in cfg.model_cost_order if m in run_models],
    "reference_model": (workflow.ceiling if workflow.ceiling in run_models
                        else run_models[-1]),
})
fresh_holdout = baseline(
    sub, holdout_tasks, bench, Path(workflow.holdout_manifest),
    dict(os.environ))
fresh_scored = (
    baseline(sub, scored_tasks, bench, Path(workflow.scored_manifest),
             dict(os.environ))
    if workflow.scored_manifest else None)
previous = model_results_from_rows(workflow.latest)
fresh = merge_partition_results(fresh_holdout, fresh_scored, previous)
latest_by_model = {result.model: result for result in previous}
latest_by_model.update({result.model: result for result in fresh})
results = [latest_by_model[model] for model in cfg.models]

Append an accepted Gate outcome idempotently. Record bundle gain against the target that entered the round, update the full scoreboard, separate held-out/scored incomplete counts, plateau history, and next report's manifests, then checkpoint complete atomically:

entry_before = next(r.holdout_passes for r in previous
                    if r.model == workflow.target)
entry_after = next(r.holdout_passes for r in results
                   if r.model == workflow.target)
applied = [list(item) for item in workflow.applied]
applied.append([
    f"round {workflow.round_n}: " + ", ".join(p.id for p in approved),
    entry_after - entry_before,
])
new_target, _ = roles(results, cfg.model_cost_order)
new_target_result = next(r for r in results if r.model == new_target)
plateau = [*workflow.plateau,
           [workflow.round_n, new_target, new_target_result.holdout_passes]]
calibration = record_bundle(workflow.calibration, workflow.round_n, approved,
                            entry_after - entry_before)
report_manifests = [workflow.holdout_manifest]
if workflow.scored_manifest:
    report_manifests.append(workflow.scored_manifest)
append_gate_outcome(Path(workflow.report_path), "accepted", approved)
workflow = checkpoint(
    state_path, workflow, "complete",
    latest=model_results_to_rows(results), applied=applied, plateau=plateau,
    calibration=calibration,
    report_manifests=report_manifests, gate_outcome="accepted",
    task_backups={})

A rate-limit pause leaves rerun_pending, so the same code resumes the same manifests. append_gate_outcome is atomic and ignores an identical block, which closes the small crash window between report append and state checkpoint.

After completion, fire a stop rule on: successful gap+floor (call stop_success with cfg.model_cost_order), six rounds, no improvement for two completed rounds with the same target, no actionable server-side finding, or user stop. Never start another round before reloading completed state and calling require_clean_tree(repo).

Step 7 — Final report (done when the report file exists and the user has its path)

Start by running the reload block from the "Round state" section, exactly as every round does — including when this step follows Step 6 in the same session, and including when Step 5 sent you straight here. It is what defines cfg, bench, manifests, tasks_dir, holdout_dir, tasks, before, applied and remaining_findings for this step. tasks comes back as the full 60 with any bumps the rounds made, because the block loads both task directories from disk.

applied is one (bundle description, delta in that round's target's held-out passes) pair per bundle that stayed in the repo; remaining_findings is the failure clusters no server-side change can address. Do not rebuild either from the last round alone, and never from memory — the final report is the whole optimization's record.

Then run one fresh baseline over the full task set for every model, including the ones that sat out the middle rounds, so the after column is complete and current. Rebuild first: a rejected bundle may have been compiled before its source commit was reverted, and an ignored build artifact otherwise survives that Git revert. For a URL target, have the user confirm the current local server was rebuilt/restarted at this point.

from whetstone_mcp.guardrails import run_test_suite

if cfg.build_command:
    built, build_log = run_test_suite(
        cfg.build_command, cfg.server_cwd, cfg.run_timeout_s)
    if not built:
        raise RuntimeError(f"final baseline build failed: {build_log}")
after = baseline(cfg, tasks, bench, manifests / "final.jsonl", dict(os.environ))
md = render_final_report(before, after, applied, remaining_findings)
final_path = state / "reports" / "final.md"
tmp = final_path.with_suffix(".md.tmp")
tmp.write_text(md)
os.replace(tmp, final_path)
workflow = checkpoint(state_path, workflow, "complete",
                      latest=model_results_to_rows(after),
                      report_manifests=[str(manifests / "final.jsonl")])

This last full re-run gets the same RateLimitPause handler: pause, tell the user the stated resume time, and finish the final report after re-running this step — final.jsonl carries everything already measured.

Present it to the user per references/report-template.md, and give them the branch names carrying the applied changes so they can merge what they want.

For a non-git target, there are no branches to hand over: write the patch from the Step 1 copy instead, using the <initial> commit you kept there.

cd <target>/.whetstone-mcp/worktree
git diff <initial>..HEAD > ../reports/whetstone-mcp.patch

Give them that file's path, say that their own target directory was never modified, and that git apply (or patch -p1) against the original applies everything the rounds kept.