shimo4228/llm-as-judge

LLM-as-Judge — Checks as Evidence, Holistic Verdict, No Scores

Core rule in one line:

Collect evidence with binary Yes/No checks, let the LLM issue one named holistic verdict, and never aggregate the answers into a score.

Why not rubric scores

• Irreproducible. Numeric scores on the same input drift between runs (3 vs 4 out of 5). LLMs are bad at independent per-dimension scoring — they get pulled by the overall impression, and central-tendency bias compresses scores into a narrow band that straddles any threshold you pick.
• Dilution. A sum or satisfaction ratio converts one fatal defect ("the referenced file does not exist") into a small deduction. One dominant No must be able to decide the outcome alone.
• Unexplainable. Nobody can say why something is a 3.5. A No answer to a concrete question explains itself and doubles as the improvement item.

Rubrics were invented to structure human evaluation, where assessors can score dimensions independently. For an LLM, invert the design: force the checks it tends to skip, and leave the judgment holistic — that is what it is good at.

The three principles

Principle	Do	Don't
① Binary checks	Decompose criteria into Yes/No questions with 1-line evidence each	"Rate specificity 1–5"
② Named holistic verdict	Pick exactly one verdict from a fixed enum, judging the whole	"Total 12 points → pass"
③ No aggregation	Enumerate the No answers as the verdict's rationale	Use the Yes-ratio as a quality metric

① Binary checks

Ask "is there a runnable command example? Yes/No", not "how specific is this?". Good binary questions are verifiable (the text settles them black-or-white) and evidentiary (a No names what is missing). Require one line of evidence per answer — a file read, a path check, a measured command output.

② Named holistic verdict

Holistic judgment — one conclusion from the whole, without passing through per-dimension numbers — is the LLM's strength. Design the verdict enum so each value maps 1:1 to a next action, e.g.:

• Library audit: Keep / Improve / Update / Retire / Merge into [X]
• Save-or-drop gate: Save / Improve then Save / Absorb into [X] / Drop

Prefer this over bare accept / reject: downstream code can dispatch on the enum directly. Verdicts that name a target (Merge into [X], Absorb into [X]) structurally forbid vague judgments like "overlaps with something".

③ No aggregation

Two rules replace the score:

1. Enumerate every No in the verdict rationale (hidden Nos breed verdict drift between runs).
2. A dominant No decides alone. If a fatal-class question is No (nonexistent reference, ungrounded claim), lean to the failing verdict even when everything else is Yes. Never let averaging dilute it.

Verdict pressure-test

Before finalizing a non-passing draft verdict, make the judge attack it:

1. Generate 1–3 atomic Yes/No questions that try to refute the draft verdict (e.g. draft = "Update: CLI flag retired" → "does the flag still appear in current --help output? measure it").
2. Answer each with one line of evidence.
3. Refutation holds → fall back toward the passing verdict.
4. Defect confirmed → the binary-screen No answers become the improvement list as-is.

Discipline: one question tests exactly one verifiable claim (atomicity); questions seek disconfirmation, not affirmation. After a fix, re-judge with the same question set, once only — regenerating questions makes it impossible to tell whether the fix worked or the bar moved.

Scale the machinery, not the principles

	Single-draft gate (N=1)	Library-scale audit (tens of items)
Pressure-test questions	Unconditional, 3–5 per draft	Non-passing candidates only, 1–3
Rationale	Cost is negligible	Generating for every item is mostly waste

Don't let deterministic checks ride on judgment

The judge's attention is a diluting resource. A controlled comparison on a 73-item skill library (2026-07) made this concrete: an everything-in-one-context pass returned all-Keep, while fresh-context small batches with unconditional reference verification surfaced 12/73 non-passing items — half with deterministic evidence (404 links, deleted files, retired CLI flags), including passing verdicts previously issued for two files that did not exist on disk. A dedicated overlap probe reproduced the set-level view (0 genuine duplications across 17 candidate clusters) without needing all bodies in one context.

Design consequences:

• Existence before judgment. Deterministically checkable claims (paths exist, flags current, URLs live) get code-owned, unconditional pre-checks — ls, --help, fetch — before the LLM sees the item. Never let a judge issue a passing verdict on a file that is not there.
• Conditional verification triggers degrade. "Verify if it looks stale" is banned phrasing: the trigger itself is a diluting judgment, and in a loaded context it decays to "never verify".
• Narrow, dense contexts for per-item judgment; light sweeps for set-level judgment. Split by the property being checked, not by context length.
• Before handing anything to the judge, ask: is this actually judgment, or is it ls?

Judge prompt template

You are a quality evaluator for {artifact type}. Evaluate as follows.

## Step 1: Binary checks (all mandatory)
Answer each with Yes/No plus one line of evidence.
- Q1: {verifiable question, e.g. is there a runnable code example?}
- Q2: {verifiable question}
- Q3: {verifiable question}
- Q4: {verifiable question}

## Step 2: Item-specific refutation questions (only if the draft verdict leans failing)
Pick one draft verdict, generate 1–3 Yes/No questions that try to refute it,
and answer each with one line of evidence.

## Step 3: Verdict
Choose exactly one. Output no scores or points.
- {verdict_1}: {meaning and next action}
- {verdict_2}: {meaning and next action}
- {verdict_3}: {meaning and next action}

Enumerate every No-answered question in the rationale.
If any dominant No ({domain-specific fatal condition}) is present, choose
{failing verdict} even when everything else is Yes.

## Output format
Follow the JSON schema below and output **JSON only**
(no surrounding prose or Markdown).

Output schema:

{
  "verdict": "Improve",
  "evidence": [
    { "question": "Runnable code example?", "answer": "No", "detail": "steps are prose-only, zero commands" },
    { "question": "Referenced paths exist?", "answer": "Yes", "detail": "ls confirmed all 3 paths" }
  ],
  "pressure_test": [
    { "question": "Are prose-only steps reproducible as-is?", "answer": "No", "detail": "step 3's arguments are ambiguous" }
  ],
  "reason": "Dominant No on actionability; adding a worked example to step 3 would reach the passing bar"
}

verdict is an enum — downstream code dispatches on it directly; the No items in evidence are the improvement list handed to the next stage.

Related

• Code-LLM layering (AKC ADR-0008) — the architecture-level "LLM judge + Code enforce" pattern (judgment never mutates state directly); this skill designs the judge's inside. Not an installed skill: it lives in the AKC repo (and as a public skill repo that harness-sync publishes to), so cite the ADR rather than a skills/ path.
• skill-stocktake — worked implementation at library scale (binary screen → pressure-test → holistic verdict, deterministic pre-pass, overlap probe).
• learn-eval — worked implementation at N=1 (unconditional dynamic questions, same-question-set re-judge).
• skill-creator §4 — creation-time draft gate: one fresh subagent without Bash, named verdict Publishable / Fix / Drop, same-question re-judge once.

References

• BinEval — "Ask, Don't Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvement" (arXiv:2606.27226). Atomic yes/no decomposition with failed questions wired into improvement feedback; the dynamic refutation questions and "No answers = improvement items" path follow it. Its own limitations motivate principle ③: decomposition is less reliable on subjective, holistic quality dimensions, and the fraction of satisfied questions need not map linearly to quality.
• Same checklist-evaluation lineage: CheckEval (arXiv:2403.18771), TICK (arXiv:2410.03608), FActScore (arXiv:2305.14251), UniEval (arXiv:2210.07197).