Elites Skills Architect

Turns a rough idea into a skill that works: discovery, spec, validation, test runs, human review, description tuning and packaging. Every claim about a skill is backed by a script or a graded run, not by reading it once and feeling good about it.

Where to start

Eight stages. Find where the user is and start there:

Discovery: get a concrete problem and the facts needed to write the spec
Flag problems: too simple, too broad, overlaps, contradictions
Categorise: document creation, workflow automation or MCP enhancement
Build and validate: frontmatter, body, folder, evals, scripts/validate_spec.py
Test and review: run the evals, show the outputs, grade them, pressure-test any rule the skill enforces
Improve: fix the pattern behind the feedback, not the single case
Description tuning: trigger tests and scripts/optimize_description.py
Package: scripts/package_skill.py

Adjust your pace to the user. Someone who says "MCP server" and "frontmatter" unprompted wants speed. Someone who says "I want Claude to do my invoices" wants to be guided one question at a time. Plain English, no jargon they did not use first.

All scripts are standard-library Python 3.9 or newer, run from the skill root:

cd <path-to-elites-skills-architect>
python3 scripts/validate_spec.py <skill-folder>

Stage 1: Discovery

Read references/discovery-questions.md for the question bank and the follow-ups for weak answers. Ask in batches of two or three, build on the answers, never dump the list.

Batch 1, problem: the one outcome the skill must produce; 5 to 10 phrases users would really say; what the skill must NOT handle. Batch 2, data: inputs and their format; what the user holds when it worked; what commonly goes wrong with the input. Batch 3, workflow: order of steps; dependencies; if/then paths; what happens to steps 1 and 2 when step 3 fails. Batch 4, integrations: MCP servers, APIs, Claude built-ins, auth and rate limits. Batch 5, resilience: the three most likely runtime failures; the message on total failure.

Stop asking when you can do all four without guessing: write the description with real trigger phrases, list every step in order, name three failure modes with their handling, and explain the skill well enough for another developer to build it.

If the user already has the knowledge written down (a handbook, API docs, a style guide, saved web pages), import it instead of retyping it:

python3 scripts/import_sources.py <files-or-folders> --skill <skill-folder> --dry-run

It turns Markdown, text and saved HTML into reference files, splits long ones at headings, writes a sources-index.md next to them, and prints the "read X when Y" lines for the body. It never downloads anything; save the pages first.

Stage 2: Flag problems before you build

Too simple: Claude does it already. A skill earns its place when there is domain logic or a format to enforce every time. Ask whether there is.
Too broad: several unrelated workflows. Suggest a split; small skills trigger more reliably. Ask which one comes first.
Overlap: close to docx, pptx, xlsx, pdf or frontend-design. Say so and ask whether to extend or to build something different.
Impossible or unsafe: say so plainly and offer the nearest feasible version.
Contradiction: "fully automatic" earlier, "ask me first" now. Resolve it before writing a line.

Stage 3: Categorise

Read references/categories-and-patterns.md. Pick one:

Category 1, document and asset creation: makes files with Claude built-ins. Discovery batches 1 and 2 are enough.
Category 2, workflow automation: ordered steps across tools, validation gates, partial failure or rollback. All five batches.
Category 3, MCP enhancement: wraps an existing MCP server with judgement and defaults. Batches 1, 2 and 4.

Tell the user the category and why. Blends are fine; pick the primary one for the description and folder layout.

Stage 4: Build and validate

Read references/worked-example.md once before generating anything, so your output matches that level of detail. Use references/spec-output-template.md as the skeleton.

4a: Frontmatter

Read references/yaml-rules.md. What the validator checks (errors unless marked):

name: kebab-case, no "claude" or "anthropic"; matches the folder name (warning, error with --portable)
description: 1 to 1024 characters; action verb, what it makes, when to use it, 3 to 8 trigger phrases in double quotes, a negative scope (the length is an error; verb, phrases and negative scope are warnings)
no angle brackets, tabs, smart quotes or dashes other than the hyphen
no apostrophes in the description (warning; some upload parsers have rejected them)
tags as one string, tags: "a, b, c", never [a, b, c] (warning); quote metadata values that look like numbers, version: "1.0"

Make the description pushy. In our tests Claude leaves a skill unused more often than it picks it wrongly, so add "also triggers when someone mentions ..." for the indirect requests.

4b: Body

At most 500 lines, imperative, and it explains why, not only what. Order: one-line purpose; when to use and when not; numbered steps with a check after each; two or three examples (phrase, actions, result); error handling with cause and response; short best practices. Move anything bigger into references/ and point at it with "read X when Y".

4c: Folder

skill-name/
  SKILL.md
  evals/evals.json        test cases, see references/schemas.md
  scripts/                deterministic logic the model should not re-derive
  references/             deep material loaded on demand
  assets/                 templates and samples

No README.md inside the skill folder; documentation lives in SKILL.md and references/.

4d: Test plan and evals

Write 5 or more should-trigger queries and 5 or more should-not-trigger queries. Good queries are specific and sound like a person: "ok kicking off the checkout redesign monday, set up the sprint in Linear, 2 weeks, frontend team". The best negatives share words with the positives but want something else: advice instead of action, reporting instead of creation.

Write 2 or 3 evals into evals/evals.json (format in references/schemas.md). Each expectation is one checkable sentence. Show them to the user before running anything.

4e: Validate

python3 scripts/validate_spec.py <skill-folder>
python3 scripts/validate_spec.py <skill-folder> --json     # for tooling
python3 scripts/validate_spec.py <skill-folder> --strict   # warnings fail too
python3 scripts/validate_spec.py <skill-folder> --portable # for agents other than Claude

It checks the frontmatter rules, body length, the sections above, placeholders left behind, evals.json, pressure-evals.json when present, and that every scripts/... or references/... path the body mentions exists. Fix every error. Never show a spec that fails. --portable adds the open Agent Skills spec rules (folder name must match, metadata values must be text) for skills that will also run in other agents.

Stage 5: Test and review

One sequence; do not stop halfway.

5a: Run the evals

Workspace layout (full tree in references/schemas.md):

<skill-name>-workspace/iteration-1/eval-<id>/<configuration>/run-<k>/outputs/

Write eval_metadata.json into each eval-<id>/ folder. Then run each eval:

With subagents (Claude Code, Cowork): start the with-skill and without-skill runs for every eval in the same turn. Each executor reads the skill, does the prompt, saves files into outputs/, writes outputs/user_notes.md for anything it was unsure of, and outputs/metrics.json. Save timing.json from the task notification at once; the numbers are not kept anywhere else.
Without subagents (claude.ai): read SKILL.md and do each prompt yourself, one at a time, into the same folders.

5b: Draft assertions while runs are going

Add measurable expectations to eval_metadata.json and evals/evals.json. Explain each one to the user in a sentence. Skip assertions for taste; a person judges those.

5c: Show the outputs before you judge them

python3 eval-viewer/review.py <skill-name>-workspace/iteration-1 --skill-name <name>

The page opens in the browser, works offline, and saves feedback to feedback.json in the workspace as the user types. In Cowork, or when a server is not possible, add --static review.html; feedback then downloads as a file. Open the viewer first and look at the outputs yourself second. Do not change the skill before the user has reacted.

5d: Grade and benchmark (subagents)

Grade each run with agents/grader.md. Use a different model from the one that ran the eval when you can; a model is a poor judge of its own work.

Aggregate:

python3 scripts/aggregate_benchmark.py <skill-name>-workspace/iteration-1 --skill-name <name>

Reopen the viewer with --benchmark <iteration>/benchmark.json for the Benchmark tab. For patterns the averages hide, run agents/analyzer.md job 2.

5e: Pressure test (skills that enforce a rule)

When the skill's value is a rule that must hold (a format, a check, an approval step), read references/pressure-tests.md and write 3 to 5 scenarios into evals/pressure-evals.json: a concrete situation, two or more pressures (deadline, a boss, the client asked, hours already spent), and a forced choice that includes a tempting compromise.

python3 scripts/pressure_eval.py --skill-path <skill-folder> \
  --scenarios <skill-folder>/evals/pressure-evals.json --runs 3 --model <model-id> \
  --output <skill-name>-workspace/pressure-results.json \
  --report <skill-name>-workspace/pressure-report.md --dry-run

Drop --dry-run after the user approves the call count. Each scenario runs without and with the skill. "skill does not hold" means the report holds the excuses Claude gave: counter each one in the skill (name it, say why it is wrong, say what to do), then run again. "passes, but the baseline passes too" means the scenario is too easy to prove anything.

Stage 6: Improve

Find the pattern behind the feedback and fix that; three test prompts are not the world. Read the transcripts, not only the outputs, and cut instructions that made the model waste steps. If you are writing ALWAYS or NEVER in capitals, stop and explain the reason instead; a model with the reason goes further than a model with the rule. When every run wrote the same helper, put it in scripts/ and say so. Beware a skill that passes its three evals and nothing else.

Then: apply the changes, rerun all evals into iteration-2/, open the viewer with --previous-workspace iteration-1 so the user sees old and new side by side, read the feedback, repeat. Stop when the user is happy, the feedback is empty, or progress has stalled.

For a rigorous old-versus-new call, use agents/comparator.md (a blind A/B judge with a swap check) and then agents/analyzer.md job 1 to explain the result. Optional; the review loop is usually enough.

Stage 7: Description tuning

The description is what makes Claude pick the skill. Tune it once the skill itself works.

7a: Trigger queries

Write 20 queries to evals/trigger-evals.json: about half should trigger, half should not. Include casual speech, file names, personal context. The valuable negatives are near misses. Show the list to the user and change what they ask to change; bad queries produce a bad description.

7b: Count before you spend

python3 scripts/trigger_eval.py --skill-path <skill-folder> --eval-set <skill-folder>/evals/trigger-evals.json --dry-run

Every run is a claude -p call. Tell the user the number and get a yes.

7c: Measure

python3 scripts/trigger_eval.py --skill-path <skill-folder> \
  --eval-set <skill-folder>/evals/trigger-evals.json \
  --runs-per-query 3 --workers 4 --model <model-id> \
  --output <skill-name>-workspace/trigger-results.json --verbose

The script installs the skill in a throwaway project and runs Claude there with only that project's settings, so nothing touches the user's real project and their other skills cannot interfere. Read the summary: missed queries (false negatives) mean the description is too narrow; wrongly fired ones mean it is too wide. A query whose runs all errored is reported as unknown, never as a pass.

7d: Optimise (Claude Code only)

python3 scripts/optimize_description.py --skill-path <skill-folder> \
  --eval-set <skill-folder>/evals/trigger-evals.json --model <model-id> \
  --max-iterations 4 --results-dir <skill-name>-workspace/description-runs --verbose

It splits the queries once into train and held-out sets, evaluates, asks Claude for a better description using only the training failures, validates the candidate with the same rules as validate_spec.py, and repeats. The winner is the iteration with the best train score; ties go to the shorter text. The held-out score is reported but never used to choose, so it stays an honest check: if it is much lower than the train score, the description is tuned to the training queries. It writes results.json, report.md and best-description.txt, and exits 0 only when the winner passes every train query. On claude.ai, read the failures and rewrite by hand instead.

7e: Apply

Paste the winner into SKILL.md, show the user before and after with the scores, and run scripts/validate_spec.py again.

Stage 8: Package

python3 scripts/package_skill.py <skill-folder> --output dist/
python3 scripts/package_skill.py --inspect dist/<skill-name>.skill

Packaging validates first and refuses on errors (and on warnings unless --allow-warnings). The archive is reproducible and prints a SHA-256. Give the user the path, or the file itself where the platform allows.

Examples

Example 1: new skill from a vague idea

User says: "I want a skill that does our Linear sprint setup from a short brief."

Actions: discovery batches 1 to 5 (category 2); flag that "modify existing sprints" is out of scope; write the spec from the worked example; create the folder and evals/evals.json; validate_spec.py passes; run two evals, open the review page, collect feedback; fix the step the executor improvised on; rerun; write 20 trigger queries, dry run, 60 calls approved, trigger eval 18/20; optimise to 20/20; package.

Result: sprint-planner.skill plus a workspace with every run, grade and review.

Example 2: skill that does not trigger

User says: "my skill is not triggering reliably, help me fix the description"

Actions: read the skill; validate_spec.py shows one quoted phrase and no negative scope; write 20 trigger queries with the user; dry run; measure (11/20, 7 misses); optimise; apply the winner (train 12/12, held-out 7/8); validate; show before and after.

Result: a new description in SKILL.md and description-runs/.../report.md.

Example 3: upload keeps failing

User says: "claude.ai rejects my SKILL.md and I cannot see why"

Actions: run the validator; it reports a smart quote on line 4 and an apostrophe in the description; fix both; validate again; package.

Result: a clean .skill file and the two findings explained.

Error handling

The user cannot state the problem: ask for the last time they did it by hand, step by step, and build the problem statement from the story.
The idea is too simple: say Claude does it already and ask what logic or format must hold every time. If nothing, advise against a skill.
The idea is too broad: split into two or three skills, pick the first.
The user contradicts an earlier answer: quote both, ask which stands.
validate_spec.py fails: fix the named finding and rerun. Do not present a failing spec.
Description over 1024 characters: cut filler first, then context, then down to the five strongest phrases. Never cut the verb, the catch-all or the negative scope.
Body over 500 lines (the validator counts): move the largest section to references/ and leave a pointer.
Trigger runs all error: check that the claude CLI is on PATH and the model id is valid; the script reports those queries as unknown, not passed. A 401 from headless claude -p while the interactive CLI works means the login is not reaching headless calls: run claude setup-token and export the token as CLAUDE_CODE_OAUTH_TOKEN.
Eval outputs consistently wrong: read the transcripts for the moment the model went off the rails; it is usually a vague instruction or a missing reason. Rewrite that instruction.
A script refuses to run: run it with --help; every script explains its inputs, and exits 2 when an input path is missing or an argument is out of range.

Platform notes

claude.ai (no subagents): run evals yourself one at a time, present results in the conversation, skip benchmarks and the optimiser. Packaging still works.
Claude Code and Cowork: start every eval run in one turn, open the viewer before judging, use --static in Cowork.
Updating an existing skill: keep the name exactly, edit a copy in a scratch folder, validate and package from the copy.
Other agents that read SKILL.md folders (the open Agent Skills format): run validate_spec.py --portable and install the folder where that agent's docs say. The trigger, pressure and optimiser scripts drive the claude CLI only.

Reference files

File	Read when	Holds
`references/discovery-questions.md`	Stage 1	Question bank, follow-ups, pacing
`references/categories-and-patterns.md`	Stage 3	Patterns per category, blends
`references/yaml-rules.md`	Stage 4a	Frontmatter rules with good and bad examples
`references/spec-output-template.md`	Stage 4	Fill-in skeleton for the spec
`references/worked-example.md`	Stage 4	Three complete worked examples
`references/schemas.md`	Stages 5 to 7	Every JSON file the tools read and write
`references/pressure-tests.md`	Stage 5e	When and how to write pressure scenarios
`agents/grader.md`	Stage 5d	How to grade a run and critique its evals
`agents/comparator.md`	Stage 6	Blind A/B judge with swap check
`agents/analyzer.md`	Stage 6, 5d	Explain a comparison; read a benchmark

Self-test queries

Should trigger: "design a skill that automates our deployment pipeline"; "help me spec out a new skill for client reports"; "my skill is not triggering reliably"; "run evals on my skill and show me how it performs"; "package my skill for upload".

Should not trigger: "fix the bug in my Python script"; "what is a Claude skill?"; "how do I install Claude Code?"; "write a Python script to process CSV files".

Ask in your favorite AI

Documentation