grahama1970/agentic-evals

Was macht agentic-evals?

Use this skill when a normal one-shot smoke test is too weak and the task needs repeatable, evidence-backed evaluation of a skill or agent workflow.

Current Scope

This initial bundle provides a deterministic fixture runner for command-based cases. It runs each case multiple times, records stdout/stderr/exit status and duration, checks explicit expectations, and emits a machine-readable readiness summary.

This proves only the declared fixture behavior. It does not prove semantic correctness, real service integration, LLM-judge quality, or release readiness unless the fixture commands themselves exercise those live paths.

Usage

./run.sh run fixtures/agentic_eval.json
./run.sh run fixtures/agentic_eval.json --output /tmp/agentic-evals-report.json
./run.sh audit-skills ../ --output /tmp/agentic-evals-baseline-gap-report.json
./run.sh scaffold-fixture ../some-skill --output ../some-skill/fixtures/agentic_eval.json
./run.sh apply-scaffolds ../ --write --output /tmp/agentic-evals-apply-scaffolds.json
# requirements-to-journey planning over live discovery (issue #1629)
./run.sh plan-journeys --fixture fixtures/agentic_eval.json \
  --requirements requirements.json \
  --interaction-inventory discovery/discovery-inventory.json \
  --state-graph discovery/state-graph.json \
  --output journey-plan.json [--scaffold-output fragment.json] [--report-only]
# claim / evidence / regression / coverage (issues #1445-#1448)
./run.sh regressions audit ../ --output /tmp/agentic-evals-regressions.json
./run.sh regressions show . && ./run.sh regressions verify .
./run.sh coverage audit ../ --output /tmp/agentic-evals-coverage.json
./run.sh coverage show .

Claim-based readiness (#1445)

Deterministic tests prove mechanisms; real E2Es prove capabilities. A pile of passing deterministic cases must never look like readiness for a live capability that was never exercised. Readiness is therefore computed per declared capability claim, then aggregated — not from a case count.

Declare capability_claims (kept separate from the claims proof-scope prose so v2 fixtures stay valid). Each case that supports a claim tags itself:

{
  "capability_claims": [
    {"id": "ask.roundtable.mixed_provider", "description": "...",
     "criticality": "critical",
     "evidence_required": {"deterministic": true, "live_e2e": true}}
  ],
  "cases": [
    {"name": "roundtable-live", "type": "positive", "evidence_class": "live_e2e",
     "supports_claims": ["ask.roundtable.mixed_provider"],
     "command": ["bash", "run.sh", "..."],
     "expected": {"exit_code": 0, "artifacts": [{"path": "out/join.json", "json_pointer": "/settled", "equals": true}]}}
  ]
}

Evidence classes: deterministic, property_or_fuzz, fault_injected_deterministic, live_e2e, adversarial_live_e2e, human_evaluation. Per-claim verdicts: PROVEN, PARTIALLY_PROVEN, BLOCKED_EXTERNAL, FAILED, NOT_ESTABLISHED. A skill is READY only when every required critical claim is PROVEN under its own evidence requirements. When claims are declared, run's gate is the worse of the case-based and claim-based readiness, so twenty green deterministic cases with an unmet required live_e2e slot cannot reach READY. A live case supporting more than one claim must carry independent per-claim artifacts, else it counts for none of them. report["capability_readiness"] holds the per-claim breakdown.

Invariant-derived claims (never stub an answer) — operator 2026-09-11

The oai-trial was lost to one omission: the requirement "no PII reaches the output" was silently narrowed to "no PII strings reach the output." Every eval, verifier, and qualification run proved the narrowed claim green while integer-typed phone numbers passed through untouched. The client read the invariant at domain scope; we proved it at mechanism scope. RULES:

1. Claims are derived from the domain invariant, not from what the code already does. Before declaring capability_claims, write the invariant in the client's words ("no X reaches the output in ANY representation"), then declare that — verbatim — as the critical claim. A claim that describes current behavior is a stub.
2. Representation matrix is mandatory for value-bearing claims. Any claim about data values (PII, money, identifiers, secrets) must enumerate the representations the input format admits — JSON string/int/float/scientific notation, CSV quoting, SQL types — and carry at least one adversarial case per representation, each capable of failing. A value-claim without a representation matrix cannot be PROVEN.
3. A coverage seam named value-representation (criticality: critical) is required for any pipeline that transforms, filters, or redacts data values. The audit must show a capable guard at that seam.
4. Independent checkers must not share the producer's data generator. The verifier's fixtures must include representations the transform's own fixtures do not, or the independence is nominal.
5. When a representation gap ships, record it in fixtures/regressions.json (incident → retained guard) and in the value-representation seam — the same class of miss must be detectable forever after.

The anti-slop contract rejects self-serving plumbing; these rules reject self-serving assumptions — the suite that is honest, adversarial, multi-trial, and still blind because every fixture shares one invisible precondition about the shape of the data.

Real end-to-end evidence contract (#1446)

evidence_class is not inferred from a command containing run.sh or curl. A case that declares a live class (live_e2e/adversarial_live_e2e) is qualified structurally and downgraded (never silently accepted) if it:

• feeds itself fixtures//stub/mocked inputs as the boundary authority;
• does not reach a substantive production entrypoint;
• has no independent readback oracle (expected.artifacts, readback: true, or stdout_excludes) — an exit code plus the command's own success prose is not proof; or
• claims live evidence for $ask browser tau-dag/compete handlers without --execute and readback from browser/provider run artifacts such as node-receipt.json, response.md, response.meta.json, browser-tab-lifecycle.json, execution-status.json, roundtable-summary.md, or compete-scorecard.json. Compile/preflight-only $ask cases are deterministic mechanism evidence, never live_e2e.

A downgraded case becomes fault_injected_deterministic (fault on a real path) or deterministic, and the report records evidence_disqualifiers. A required live slot met only by a BLOCKED_EXTERNAL case leaves the claim non-PROVEN.

Exemptions are explicit and expiring. A claim may carry exemptions naming evidence_class, reason_code, justification, owner, and expires. A valid exemption is surfaced in the report (exempt_evidence) but never makes a claim PROVEN; an expired or incomplete exemption is ignored. No blanket eval_not_required satisfies an executable operational capability.

Incident → retained regression (#1447)

A live failure becomes permanent evidence in fixtures/regressions.json (agentic_evals.regressions.v1). Each record links the incident to the claims and seams it threatened, names the retained guard case (retained_case / retained_fixture), and carries a fail_before_fix.proof_command.

• regressions show <skill> / regressions audit <root> report: unprotected regressions (retained case missing/renamed), never-proven (possibly vacuous) regressions, stale live proof, retired regressions, and open incidents with no regression mapping.
• regressions verify <skill> re-runs each proof command and confirms the guard actually fails against the broken behaviour (exits non-zero) — a non-vacuity proof, not a self-reported flag. It exits non-zero if any active regression's guard cannot demonstrate fail-before-fix.

Fixing a bug by editing the eval expectation to match broken output is exactly what the non-vacuity proof catches. Retiring a regression requires a retirement reason.

Risk-based coverage sufficiency (#1448)

Counting cases is gameable. coverage audit/coverage show ask, per declared seam: is there at least one case capable of detecting a regression at that seam, in each evidence class the seam's risk requires? Declare seams with seam_id, seam_type, criticality, and required_evidence; tag cases with seams. A case covers a seam only when it lists that seam and carries an oracle that can fail — a bare exit 0 positive is weak_only and does not count. The audit reports covered/uncovered seams, weak-only coverage, live vs deterministic coverage, mapped incident regressions, and prioritized next evals. Coverage (a capable guard is declared) and freshness (a live proof is current) are separate dimensions; the runner does not run the seam cases here.

Fixture Contract

{
  "version": 2,
  "skill": "example-skill",
  "trials": 3,
  "proof_scope": "fixture wiring smoke",
  "claims": {
    "proves": "the declared command exits with the expected status",
    "does_not_prove": "semantic correctness, live service behavior, or full skill readiness"
  },
  "cases": [
    {
      "name": "happy-path",
      "type": "positive",
      "command": ["echo", "success"],
      "expected": {
        "exit_code": 0,
        "stdout_contains": ["success"]
      }
    }
  ]
}

Each case must declare:

• name
• type: positive, negative, or adversarial
• command: a non-empty argv list
• expected.exit_code

Optional expectations:

• expected.stdout_contains
• expected.stderr_contains
• must_exercise: list of downstream contract command substrings that must appear in the case's top-level command. Use this when a wrapper could otherwise fake success; for example an eval that claims $ticket/$project-watchdog routability must include "skills/ticket/run.sh feature", not only a helper script that prints OK.

Anti-Slop Contract (fail-closed)

A skill evaluation fixture is REJECTED at load time (non-zero exit, no run) if it is self-serving deterministic plumbing rather than real-world proof. To pass, a skill fixture MUST:

• set trials >= 2 (a single trial is not evidence);
• include at least one negative or adversarial case (an all-positive fixture is self-serving);
• include at least one real-world case: "real_world": true whose command exercises a live path (the skill's run.sh / a script / live HTTP / a test runner) and does NOT feed itself fixtures/ stub inputs;
• contain no trivial echo/constant cases that prove nothing.

Rejection message names every violation. This prevents an eval that passes trivially while proving nothing about whether the skill actually works.

Compliance tier ("eval_tier": "compliance")

A fixture that guards a compliance-pipeline stage declares "eval_tier": "compliance" and the runner then MANDATES the strong contract on top of the baseline (operator directive 2026-08-12, "this is a compliance pipeline and must be robustly hardened"). Such a fixture is REJECTED unless:

• a strict majority of cases are adversarial/negative (more than half, not exactly half — positive controls are the minority);
• at least one case is non-deterministic: its command samples fresh inputs each run via --samples, --seed, or a shell $RANDOM (a probe script name with a fixed key does not count);
• every non-deterministic case names --samples >= 50, so each stage's coverage is hundreds-to-thousands of assertions per run, targeting ~1000 per stage across its modes.

The declaration cannot be quietly relaxed: the compliance pipeline's own fixtures set the tier, so removing it to dodge the gate is itself a regression. tests/test_compliance_tier_gate.py pins each rule against its weakening.

Two honestly-declared exemptions bypass the gate — never valid for a real skill evaluation:

• "eval_kind": "runner_selftest" — a fixture that tests this runner itself or is a documentation example.
• "eval_kind": "scaffold" — the mechanical first-posture fixture emitted by scaffold-fixture / apply-scaffolds, which the audit still flags as needing real cases.

Readiness Mapping

Readiness is scored over required cases only ("required": false opts a case out). A required BLOCKED case cannot reach READY: an unmet precondition is absence of evidence, not evidence of success.

• READY: every required case passed every trial.
• USABLE_WITH_GAPS: at least one trial passed and at least one did not.
• NOT_READY: trials ran but no required case fully passed.
• NOT_ESTABLISHED: no cases were executed.

Fail-closed exit

run exits non-zero unless readiness is READY. A runner that exits 0 on USABLE_WITH_GAPS lets an outer CI job go green over failed cases, which is the whole failure this gate exists to prevent. Pass --report-only when you want the report without the gate.

Case outcomes

Each case reports one outcome, because these mean different things to a gate:

Outcome	Meaning
PASS	every trial met every expectation
FAIL	a defect, or a timeout, or a trial that left a child process behind
BLOCKED	a precondition was unmet; declare markers via blocked_when_stdout_contains
NOT_TESTED	no trials ran

Artifact assertions

stdout substring matching cannot express "these two receipts name the same session" or "the run left nothing behind". Declare expected.artifacts:

"expected": {
  "exit_code": 0,
  "artifacts": [
    {"path": "out/session.json", "json_pointer": "/sessionId",
     "equals_artifact": {"path": "out/detach.json", "json_pointer": "/sessionId"}},
    {"path": "out/attach.json", "json_pointer": "/phase", "equals": "attach"},
    {"path": "out/tmp.lock", "absent": true},
    {"path": "out/report.json", "min_bytes": 32, "sha256": "sha256:..."}
  ]
}

Paths resolve relative to the fixture directory. Verified artifact hashes are recorded on the trial.

Process-group teardown

Each trial runs in its own process group. On timeout the runner kills the group, then re-reads /proc and records any survivor in orphan_pids_after_teardown; a non-empty list fails the trial. A timed-out case that strands a grandchild holding a lock silently corrupts every later case in a serial run, so teardown is verified rather than assumed.

Report provenance

The report is agentic_evals.report.v2 and carries run_id, per-case case_id, per-trial trial_id, the exact argv, fixture_sha256, and repo.sha/repo.ref when available. It preserves the manifest's own proof_scope and claims instead of substituting a generic fixture-only claim, and reports live: true when the manifest declares it or any case is real_world. Reports are written atomically.

Every executed case/trial also carries execution_provenance. The closed execution_mode vocabulary is:

• exploration — candidate generation/discovery; never counts as executed capability proof.
• regression_replay — default for compatible v2 fixtures; may satisfy a claim only when no test/oracle mutation is declared or detected, and when any claim-level admitted_evidence record matches the frozen test/oracle hashes.
• test_repair — intentional locator/step/input/assertion/oracle maintenance; reports before/after hashes and requires explicit requalification before a later unchanged replay may satisfy a claim.

The runner records test_source_sha256, oracle_sha256, generated_test_lineage/generation_id when supplied, prior_test_source_sha256, prior_oracle_sha256, application/build identity, closed mutation flags (test_mutated, locator_healed, oracle_changed, expected_output_changed, etc.), and evidence_eligibility. Provider/import adapters may declare self-heal provenance through external_result or provider_result, but a provider-reported pass with mutation flags is treated as repair evidence, not readiness authority. Undeclared test/oracle hash changes fail closed as evidence-integrity errors.

Self-tests for every behavior above: fixtures/runner_selftest.json.

Remediation loop (categorize → ticket-with-depends-on → fix → re-run until green)

Status: DESIGN v2 (WebGPT-reviewed) — full contract in REMEDIATION_LOOP.md. Running a suite and reporting NOT_READY is only half a loop. The standard remediation loop turns eval failures into fixes deterministically, identically for every project:

run ALL evals to completion (campaign-frozen inputs)
  → render the active category DAG (phart-dag-chart) + validate acyclicity   [REQUIRED on failure]
  → categorize the COMPLETE failure set into stable root-cause category_ids
  → plan ticket + depends_on mutations, validate the plan, apply atomically
  → project-watchdog dispatches routable (unblocked) tickets concurrently
  → per ticket: `ask tau-dag --topology sequential`, gpt-5.5-high (creator) +
    fable-5-low (reviewer with tampering VETO) → category-green (fast check)
  → integrate → re-run the FULL suite on the merged head → fresh re-categorize
  → stop when zero categories remain, or a fingerprint/budget blocker fires

Non-negotiable rules (each prevents a named failure mode):

1. Ticket only after a complete pass — a defect class is visible only once every failure belonging to it has been seen; one-per-case floods the fixer with racing patches for one bug. (Sole exception: an incident/abort path for infra corruption, destructive behavior, credential leak, or runaway cost.)
2. Categorize before ticketing and before edges — edges are defined over categories shown to exist in THIS run.
3. Re-categorize every iteration — a landed fix reshapes the failure surface; consume the exact frozen report, never "latest".
4. The INTEGRATED full re-run is the close gate, not a category-only run and not the reviewer — keep three distinct facts: REVIEW_ADMISSIBLE → CATEGORY_GREEN (fast local check) → FULL_RUN_RECONCILED (complete suite on the merged head, category absent, no regression) = CLOSED. Reviewer FAIL VETOES closure against tampering with protected surfaces (fixtures, oracles, category maps, proof commands).
5. A phart-dag-chart visualization is REQUIRED on any failing run — it both shows the operator what failed and what blocks what, and serves as the acyclicity gate: no ticket block may apply while the active DAG is invalid.

Guardrails the contract also mandates: stable immutable category_id (≠ the mutable GitHub label) with an explicit split/merge/disappear/new-category reconcile lifecycle; a validated active-only dependency mutation plan (edges wired only to currently-active categories, sparse by default, each with a rationale); fingerprint-based oscillation/no-progress termination with campaign-frozen inputs; same-repo-only v1; and a self-modification privilege boundary (a consumer's remediation ticket may not edit /agentic-evals, /ticket, /project-watchdog, or the schema).

Each consumer project supplies only fixtures/agentic_eval.json and a category_map.v1 (stable category_ids + active depends_on edges, tied to its capability_claims/seams). The shared machinery — remediate, categorize, category-dag, run --only-category — lives here. It composes existing /ticket, /project-watchdog, /ask tau-dag, phart-dag-chart, and ticket_closure_evidence.v1 and invents no parallel ticketing, dispatch, closure, or charting system. The memory repo's probe_failure_triage.py is the prototype this generalizes and retires.

Composition

agentic-evals composes with eval-skills: use eval-skills for the existing repository fixture schema and broad skill regression checks; use agentic-evals when the evaluation needs repeated trials, trajectory-oriented case typing, and readiness-state output.

Use audit-skills after changing best-practices-skills eval rules. The audit does not prove per-skill behavior; it proves the repository's current eval posture by recording which skills already have fixtures, delegate to eval skills, document eval_not_required, or still emit EVAL001.

Use scaffold-fixture only as the first mechanical eval posture for a skill. A generated fixture proves wiring only until a human or maintainer adds skill-specific positive, negative, and adversarial cases.

Use apply-scaffolds to apply that first mechanical posture across all currently scaffoldable EVAL001 skills. For skills with sanity.sh or run.sh, it creates an entrypoint-backed fixture. For skills without an entrypoint, it creates a static contract-validation fixture that runs the best-practices-skills validator from the skill's fixtures/ directory. It writes only missing fixtures/agentic_eval.json files unless --force is passed and emits a JSON receipt. This reduces missing eval posture; it does not establish semantic coverage.

Regression Fixture Pattern

When a live incident exposes an agent-troubleshooting failure, add or strengthen the affected skill's committed fixtures/agentic_eval.json instead of leaving the lesson only in chat. The case should name the failure code, exercise the real skill entrypoint, script, or test runner, and assert the recovery wording or receipt fields an agent must see.

Example pattern for browser transport incidents:

• type: "adversarial" for stale sockets, stale tab bindings, lock contention, missing native host dependencies, or provider payload mismatch.
• real_world: true when the command invokes run.sh, a real script, live HTTP, or the production test runner without feeding itself fixtures/ stubs.
• expected.stderr_contains or expected.stdout_contains should include the stable failure code such as stale_socket_no_listener, not only a generic timeout or nonzero exit.

This keeps agentic evals tied to the operational mistake future project agents need to recognize.