Belweave/belay

What does belay do?

You reach a goal in many small steps. A check scores every step with no human in the loop. scripts/belay.py keeps the score: you edit, it runs the check, commits a win or undoes a loss, and writes the log. You never write a score down yourself, and you never decide a step "basically" passed.

Every goal becomes the same thing: a command that prints a number. Faster is a time or a count. An audit is the number of failing tests, one per finding. A port is the number of outputs that match. A UI is a judge's score.

How to work:

• The user starts a climb and leaves. Don't stop to give updates. They read three things: the card, a question only they can answer, and the report. Ask with the harness's question tool if it has one.
• State lives on disk, not in chat. The card, the log, your notes, and the last check output are under .belay/<name>/. After a compaction or a crash, status gives it all back in one call.
• Never wait with a sleep loop. Run long commands in the background and plan the next step while they run.
• BELAY below is the absolute path of scripts/belay.py next to this file. Write the full path into each command. Run it from inside the target repo. Paths and commands you pass are relative to the repo root.
• The climb gets its own branch, climb/<name>. An undone step resets the tree and deletes untracked files git doesn't ignore. If anyone else is working in this checkout, climb in a separate worktree. Pass --here when you are already on a branch made for this work.

1. Resume before you start

Run python3 "$BELAY" list. If a climb for this goal exists, switch to its branch, run status <name>, and continue at step 4. "Continue" after a crash or a new session means this. Don't start over.

2. Build the check

The climb is only as good as its check. A check that measures the wrong thing spends the whole run making the wrong thing better. Get a rough check running first and sharpen it later with relock.

Ask: what would I run to see that the goal is met? Make that command print a number. Use the strongest form that fits: an exact match or test count, then a count of the work (queries, round trips, bytes), then a clock, and a model judge only when nothing but taste can tell.

Goal	Score
Bug or feature	Failing tests written from the spec, not from your fix
Audit, stabilize, "make X work"	Failing tests, one per finding
Speed or size	A count of the work behind the time, or the time
Port or clone	Outputs that match the real tool
Several goals at once	Progress toward the worst bar
Look, feel, writing	judge.py with a rubric and good examples

Recipes for each are in references/checks.md. Read the part that matches your goal before you write the check.

Use what the repo has. --score '<regex>' reads the score from any test runner's output, like --score '(\d+) failed'. It takes the last matching line and adds up the numbers it captures. A passing run with no match scores 0. Without --score, the last line of the check's stdout must be the bare number. Repeat --check to run several commands at once; their scores add. Add --repo <path> for each other checkout the climb edits. A step is then kept or undone in all of them together.

Prove the check before you trust it:

• It can fail. Break the code on purpose and see the score get worse. Restore with git checkout -- <file>.
• It can see a win. A small real improvement must move the score.
• It is steady. Run it 3 times. If the score drifts with no code change, replace the clock with a count, or use --runs 3, a --min-gain above the drift (a percent like 3% when the score will shrink a lot), and --paired.
• It can't be gamed. Vary the input each run and compare with a reference. Try one cheat by hand, like hard-coding the answer, and confirm the check or a guard catches it.
• It is fast. You pay the check time on every step. Get a step under 2 minutes: score a sample or a count per step, and keep the full suite or the timed run as a guard or the holdout.
• The bar is reachable. Work out the best the current design can do. If the bar is beyond it, say so on the card and propose a bar you can reach.

Protect everything the score doesn't measure:

• Guards (--guard, pass by exiting 0). List what the code you will change touches: callers, other platforms, saved data, public APIs. Each needs a guard: the tests that pass today, typecheck, lint, the build. When the score is a failing count, keep those passing tests in their own files as a guard, so a crash can't pass as "fewer failures".
• Lock (--lock) the check, the tests, the fixtures, and the holdout. try refuses to score while a locked file differs. New files under a --lock path become the first commit on the climb branch. Any other uncommitted change stops init.
• Holdout (--holdout), when the check uses samples, a benchmark, or a judge: cases the climb never trains on, kept under .belay/<name>/holdout/. Don't open them again. If the score rises and the holdout doesn't, the climb is overfitting. Stop and say so.

Audits and open goals. List what the goal covers as a checklist in .belay/<name>/notes.md. Hunt for bugs area by area. Write one failing test per real finding, from the spec or the bug report. When a later step finds a new bug, add its test and run relock --why "found <bug>". The score rises for an honest reason and the log shows why. A finding that needs the user's decision goes in the notes and the report, not in the score.

3. The card, then init

Send one message with the card and nothing else to decide:

Climb card: build-speed

Goal: cold build 2x faster, same output

Score	Median build seconds, lower is better: 47.0 now, bar 23.5
Check	./scripts/bench-build.sh, 3 runs, paired, min gain 3%
Guards	npm test, npm run typecheck
Holdout	3 of the 12 sample apps
Locked	scripts/bench-build.sh, test/, samples/
Limit	none (about 2 min a step)

Needs your answer: none

Reply go, or change a line.

Fill every line with your best choice. If the user gave a goal with no bar, propose one that is ambitious and checkable, like "2x faster" or "every sample matches". Wait for "go", which approves the card, the branch, and its commits. Skip the wait and put the card at the top of the report when the user already said to run without asking, or when the goal is small and the bar is not in doubt. After that, ask nothing you can find out or decide yourself.

python3 "$BELAY" init build-speed --goal "cold build 2x faster" \
  --check ./scripts/bench-build.sh --direction min --target 2x \
  --runs 3 --paired --min-gain 3% --guard "npm test" --guard "npm run typecheck" \
  --holdout ./scripts/bench-holdout.sh --lock scripts --lock test \
  --lock .belay/build-speed/holdout

An audit over a backend and a frontend:

python3 "$BELAY" init billing-audit --goal "every billing finding fixed" \
  --check "pytest tests/billing -q" --check "cd ../web && npx vitest run billing" \
  --score '(\d+) failed' --direction min --target 0 --repo ../web \
  --guard "pytest tests/legacy -q" --guard "cd ../web && npx tsc --noEmit" \
  --lock tests/billing --lock ../web/src/billing/__tests__

init makes the branch, measures the baseline, and refuses to start if a guard already fails. Fix what it warns about before the first step. Set --max-hours or --max-steps only when the user names a limit. Without one, the climb runs until the bar is met or it is stuck.

4. Climb

Each step is one idea, one change, one try:

1. Pick the idea. Don't retry one the log already undid unless you can say what is different now. If judges are in use, read .belay/judge-notes.md.
2. Make one change for one reason and guess its effect. Put the guess in the idea: "cache the parser (expect -10%)". When a guess is far off, your model of the code is wrong. Profile before the next step.
3. python3 "$BELAY" try "<idea>". It commits a win or resets the tree. Either way, move on. The step's full check output is in .belay/<name>/last.txt, and the best step's in best.txt. Read those instead of running the check again.
4. While a check runs, don't edit the tree. Only one belay.py command that changes the climb runs at a time. If it says another is still running, wait for it.
5. Every 5 steps, update .belay/<name>/notes.md in a few lines: what works, what doesn't, what to try next. The next session resumes from it.

If you believe in an idea the check undid twice, the check can't see it. Make the check finer and relock. Never edit a locked file to make a step pass.

Never cheat the check

Each of these raises the score and makes the work worse. Each is a failed climb even if try keeps it:

• Special-casing the inputs the check uses, or hard-coding or caching expected outputs.
• Skipping, weakening, or deleting tests, or catching and hiding errors.
• Making the benchmark do less work instead of making the code faster.
• Using relock to drop or loosen a hard case so the bar is met. Relock only makes the check more correct. If the bar can't be reached, report the floor you measured.
• Reading the holdout.

A real win makes the same code better on inputs the check has never seen. If the only way forward is one of these, the check is wrong. Say so.

Keep reevaluating

The score only stands in for what the user wants. Check that it still does:

• Every 10 kept steps, and at the bar: look at the real thing yourself. Open the app, read the output, view the screenshot. Run the full test suite and holdout. If the score and the real thing drifted apart, fix the check and relock --why "...".
• Something broke that no guard caught: find the step with git bisect run <failing command>, then repair it with try --fix "<what broke>", which keeps a change that holds the score. If the only repair gives up that step's gain, git revert the step and relock --why "reverted step N: broke X". Add the guard that was missing.
• Before the report: review git diff <start>..HEAD as one change, with fresh eyes. Look for removed tests, swallowed errors, a feature turned off, a changed default, and data or API changes. Repair with try --fix, then run the check and holdout --why final once more so the report quotes measured numbers.

5. When it stalls

try and status print PLATEAU after 5 steps with no gain. Stop trying ideas and find the blocker. Measure before you guess:

• Where does the time or the error count actually come from? Profile it.
• Is the check flaky, too coarse to see small wins, or blind to the problem?
• Is there a bigger idea the small steps can't reach?
• What tool, access, data, or compute would change the next ten steps?

Log it with python3 "$BELAY" review "<what you found>", which quiets PLATEAU for 5 steps. Fix what you can: a faster check, a finer metric, a different approach. Ask the user only for what only they can give, like access or money, and only after every other idea is spent.

6. Stop and report

Stop when any of these is true:

• try prints BAR REACHED. It runs the holdout for you. A visible win with a flat holdout is not a win.
• status prints STUCK: two blocker reviews with no gain between them.
• try refuses with "Budget used".
• The holdout disagrees with the score.
• The next step needs something only the user can give.

Then do the final review above, delete .belay/<name>/data/, and send the report. Make it readable in ten seconds:

Climb build-speed: bar reached

47.1s → 18.7s, 2.5x faster (bar 23.5s, reached at step 41)

Holdout	51.0s → 20.2s, moved with the score
Steps	19 kept of 52
Safety	Guards passed on every kept step. Full suite and final review clean
Branch	climb/build-speed, 19 commits

What worked

• Incremental type cache: −12s

Tried and undone

• swc swap: broke 3 guards

Your move

• Review: git log --oneline 1a2b3c4..climb/build-speed

The headline is one of: bar reached, stuck, budget spent, holdout disagrees, needs you. Quote numbers the script measured, never estimates. If you changed the check or the bar along the way, say so in the headline. Biggest wins first. At most 3 bullets a section. Skip empty sections.

If the user then wants more, set a harder bar for the same claim with raise 2x --why "..." and keep climbing. Never switch to a new goal.

Subagents

If the harness can start subagents, use them for parallel reading during setup and for fresh eyes on the blocker review and the final review. Ask for conclusions, not file dumps. Only you write in the climb tree, because try resets it. A helper reads, or drafts a patch in its own worktree for you to apply and try. Without subagents, do it inline.

Commands

Command	Does
init <name> --goal "..." --check "..." --direction min|max	Baseline, branch, lock. Options: --check (repeat), --score '<regex>', --target (number or 2x), --guard (repeat), --lock (repeat), --holdout, --repo (repeat), --runs, --paired, --min-gain, --timeout, --max-hours, --max-steps, --here
try "<idea>" [--fix]	Score the edit, then keep or undo it. --fix keeps a repair that holds the score
status [name]	Score, last steps, notes, PLATEAU or STUCK
relock --why "..."	Accept a changed check and measure again
holdout --why "..."	Run the holdout now
review "<found>"	Log a blocker review
raise <bar> --why "..."	Set a harder bar after reaching this one
budget [--hours N] [--steps N] [--off]	Give N more hours or steps from now, or remove the limits
drop [name]	Abandon a climb whose card was wrong. Keeps its branch and puts the check files back for a new init
list	Climbs in this repo

Every command takes --name <climb> when the repo has more than one climb.