wptoux/harness2

Harness² — Recurring Task Harnessing

Adapted from Harness²: Recursive Agent Harnessing for an Open World, cropped to the case where one execution has already been accepted and the goal is to repeat it.

Overview

Succeeding once is not the same as being repeatable. The accepted run is your only trustworthy reference point: it has a real deliverable, a real trajectory, and the places the user personally edited.

The object of optimization is the next run's process, not this run's deliverable. So the output is not a better report — it is a harness, an SOP, and a regression checklist.

When to Use

• The user says "we'll do this every week from now on" or "let's lock this process down"
• The same kind of task has been done at least once, and that result was accepted
• The process lives in someone's head: who runs which script first, which step needs a human check, where the traps are
• The same class of task keeps failing the same way: missing source attributions, inconsistent definitions, constant rework

When NOT to Use

• One-off task → just do it
• A real grader, unit test, or computable acceptance criterion exists → use it; it beats behavioral contrast by a wide margin
• The user only wants a format or definition changed, and has said which → edit the harness directly, skip the contrast loop
• The process is already stable, and cost or latency is tight

What a harness is

Everything around the model that can be edited. Four are enough to run; eight is the full set:

Component	Role	Minimum
system prompt	role, output contract	✅
tool guidance	when to use each data source, how to fill arguments	✅
skills	reusable judgment and procedures	✅
guardrails	hard constraints (never invent numbers, always cite sources)	✅
memory	domain notes carried across instances
scripts	deterministic work (conversion, dedup, aggregation)
subagents	verification or parallel retrieval split out
plugins	middleware (force a self-check every N steps)

A harness must exist as files to be diffed and reused. "We usually do it like this," left inside a chat, cannot be solidified.

The loop: Propose → Probe → Compose

0. Freeze the baseline

Archive the last run in full: harness snapshot, complete trajectory (tool-call sequence + intermediate artifacts), final deliverable, the user's edit diff, acceptance criteria, data snapshot or time window.

Places the user personally edited are high-confidence defect locations — more reliable than any guess.

1. Propose

Read the baseline, then propose two candidates, each with one sentence naming the defect it targets:

• invariant: process defects — no cross-checking, intermediate results never persisted, failures not retried, sources unlabeled, context blown out
• domain: professional constraints of this task class — consistent definitions, currency or unit conversion, aligned time windows

You need both to separate "general disease" from "disease specific to this task class." With a single candidate you cannot attribute the outcome.

2. Probe

Replay with the same input, controlling variables: same model, same tool set, same data snapshot. Only the harness changes.

Compare two things:

1. Behavioral delta against the baseline: tool set and order, step count, repeated calls, errors and retries, whether human-patch points disappeared, whether intermediates were persisted, whether sources are fully attributed
2. Replay consistency: run it again — is the structure the same? The core metric of solidification is stability, not single-run best

Contrast shows "behavior changed." It does not show "the result is right." Never conflate the two.

3. Compose

Keep only changes that (a) still meet the acceptance criteria after replay and (b) reduce rework or human patching. Keeping the original is allowed — a change without evidence is noise.

Record every decision on one line: change | defect targeted | evidence observed | adopted / revised / dropped.

4. Solidify

Three artifacts; missing any one means it is not solidified:

• harness/ — the editable files
• SOP.md — steps, checkpoints, when a human must be asked, known traps (over ten items and nobody reads it)
• REGRESSION.md — the acceptance checklist to walk through after the next run

If a change is adopted across several rounds, promote it to a standalone script or skill.

Evidence hierarchy

user edit diff > comparison against the accepted deliverable > replay consistency > trajectory delta > intuition

Not evidence: a longer or more detailed report, "the user didn't complain this time", the model claiming it did better.

Budget

Default k=1 plus one replay. Solidification pays off over time, so amortize it across rounds: probe one candidate during each recurring run, instead of spending 3× once to run every parallel branch.

Red flags — stop when you see one

• Inheriting the previous harness wholesale without reading the user's edit diff
• Replaying with a different input, model, or data snapshot and still claiming "it's better"
• Concluding from a single replay without ruling out randomness
• Changing five components at once, making attribution impossible
• A 20-item SOP nobody will follow next time
• Writing a one-time lucky trick into an invariant entry

Common mistakes

Mistake	Consequence	Fix
Optimizing before archiving	baseline lost, changes unattributable	freeze the baseline first
"It reads better" as evidence	you solidify a preference, not a process	bind it to a concrete observation
Prompt-only changes, no checkpoints	humans still patch by hand next time	pair every defect with a checkpoint
SOP written as a travelogue	nobody reads it, so nothing is solidified	write decision points and traps only