Evaluator Validity Audit
Scope
Use on deterministic or model-assisted evaluators where grading defects can make incorrect answers look correct or legitimate alternatives fail.
Procedure
- Name the construct being measured: answer correctness, evidence grounding, required tool action, authority boundary, population scope, etc. Separate these dimensions.
- Map each task to reference solutions, input assumptions, evaluator logic, accepted alternative behaviors, and explicit failure codes.
- Test positive controls (genuinely correct), negative controls (wrong values, missing evidence, spoofed tool outcomes, impermissible actions), and valid alternatives (equivalent SQL, evidence paths or reasoning dispositions).
- Verify that observation fields are derived from actual runtime evidence, not model-authored labels or untrusted result metadata.
- Run controls without exposing held-out instances to the development agent. Preserve counterexamples and their original evidence.
- Classify defects as false acceptance, false rejection, ambiguous task contract, or unsupported but valid alternative. Distinguish scorer defects from model or runtime failures.
- Propose versioned changes with expected behavioral deltas; do not silently tune the grader after seeing confirmatory outcomes.
- Freeze only when the stated validity gate passes, or explicitly report outstanding limitations and withhold claims.
Output
An audited case matrix showing reference, adversarial, and alternative cases, expected vs observed reason codes, and unclosed defects.
Stop conditions
Pause confirmatory interpretation for unresolved defects that materially alter measured success. Tests passing alone do not establish validity if important counterexamples are missing.