YoursSarcastically/eval-design

Que fait eval-design ?

You are designing the measurement, not the product. The output of this skill is a decision someone can defend in a room, not a number.

Most of the time the decision is what to fix next. An eval that only ranks systems is a report; an eval that names the two things costing you the most, and proves the fix worked when you re-run it, is a development loop. Build for the second one — the failure taxonomy and the per-criterion breakdown matter more than the headline score, because those are the parts someone can act on.

The position

Four rules. They are opinions, and they are the point of this skill.

1. No vibes. "It feels better" is not a result. If a change cannot be expressed as a movement on a defined criterion measured on a fixed set, it has not been evaluated. Ship it if you like — just do not claim it was tested.

2. Every criterion must be disagreeable. A criterion two careful reviewers could not land on opposite sides of is measuring nothing. "Is the output helpful?" is not a criterion. "Does the reply state the refund window in days?" is. Every criterion in a rubric carries the specific disagreement it was written to settle — see references/rubric-design.md.

3. No LLM-as-judge by default. A judge is a system with its own error rate, and an unvalidated judge launders that error into your headline number. Humans label first, the judge is calibrated against those labels, and the agreement is published next to every result the judge produced. See references/judges.md.

4. Report the baseline or the eval is marketing. Regex, a keyword rule, the current production system, doing nothing. Roughly a third of shipped AI features lose to something that costs nothing on at least one criterion, and you want to find that out before a hiring manager or a VP does.

Two structural rules follow from these:

• Gates are floors, not weights. A safety, legal, or trust criterion does not get traded against average performance. A system below the floor is not a candidate at any price. Weighted averages hide exactly this.
• Confidence intervals or no ranking. At n=150 a 3-point difference is noise. Ranking systems inside the noise is the most common way eval work misleads the person reading it.

Workflow

Work in this order. Do not skip to scoring — every failure of an eval traces back to something skipped above it.

1. Find the decision. What changes based on the result? "Which model do we ship", "can this go unsupervised", "did the prompt change regress". If nothing changes, stop and say so — this is the single most valuable moment in the process, and the honest answer is sometimes "you do not need an eval".
2. Write the failure taxonomy first. Before criteria. Collect 30–50 real outputs, read every one, and cluster what went wrong. Criteria derived from observed failures beat criteria derived from imagination, every time. → references/failure-taxonomy.md
3. Turn the taxonomy into criteria. Each gets a weight, a disagreement, and optionally a gate. → references/rubric-design.md
4. Build the golden set. Stratified, adversarial, versioned, and small enough that a human actually labels it. → references/golden-sets.md
5. Label with humans, measure agreement. If two labellers cannot agree, the rubric is broken, not the labellers. Fix the rubric and relabel. → references/statistics.md
6. Only now consider a judge, and only calibrated. → references/judges.md
7. Run every system including a non-AI baseline, write results.json, and render. → references/scorecard-spec.md
8. Write the verdict yourself. The scorecard shows numbers; a human states the trade being made and what would reverse the decision.

Rendering

python3 skills/eval-design/scripts/render_scorecard.py results.json --md

Writes a self-contained scorecard.html (no external requests, light and dark) and a SUMMARY.md for reading in the repo. Schema: references/scorecard-spec.md.

python3 skills/eval-design/scripts/validate_rubric.py results.json

Fails on the mistakes that make a scorecard dishonest: weights that do not sum to 1, criteria with no stated disagreement, gates with no justification, missing baseline, or a claimed ranking inside the confidence interval.

When not to use this

• A one-off prompt tweak with no decision attached. Just look at the output.
• You have no users and no examples. Get 30 real inputs first. An eval built on imagined inputs measures your imagination.
• The task has a correct answer a program can check — arithmetic, valid JSON, a compiling build. Write the check, not a rubric.
• Someone wants a number to justify a decision already made. Say that out loud instead of building the eval.

Reference files

Load on demand; do not read them all up front.

File	Read when
references/rubric-design.md	Writing or fixing criteria
references/failure-taxonomy.md	Starting from real outputs
references/golden-sets.md	Choosing and stratifying cases
references/judges.md	Anyone proposes LLM-as-judge
references/statistics.md	Sample size, CIs, agreement, drift
references/scorecard-spec.md	Writing results.json
references/antipatterns.md	Reviewing someone else's eval