skill-forge-evolve

스킬 진화 및 개선

프로세스

1단계: 문제 진단

사용자에게 질문하거나 로그를 분석하여 문제 범주를 식별합니다:

범주 A: 트리거 문제

• 과소 트리거: 스킬이 필요할 때 활성화되지 않음
• 과다 트리거: 스킬이 필요하지 않을 때 활성화됨
• 잘못된 트리거: 잘못된 하위 스킬이 활성화됨

범주 B: 실행 문제

• 불완전한 워크플로우: 스킬이 완료 전에 중단됨
• 잘못된 출력: 결과가 기대치와 일치하지 않음
• 누락된 오류 처리: 실패가 정상적으로 처리되지 않음
• 성능: 너무 느리거나 너무 많은 토큰 사용

범주 C: 아키텍처 문제

• 누락된 기능: 새로운 사용 사례가 다루어지지 않음
• 규모 문제: 스킬이 너무 커서 분해가 필요함
• 상호 참조 문제: 존재하지 않는 파일에 대한 링크

범주 D: 품질 문제

• 일관되지 않은 결과: 동일한 입력에 대해 다른 출력
• 모호한 지침: Claude가 매번 다르게 해석
• 누락된 예제: 구체적인 안내 부재

2단계: 범주별 수정 적용

수정: 과소 트리거

1. 현재 설명 읽기
2. 누락된 트리거 문구 식별
3. 도메인 키워드 및 의역 추가
4. 관련된 경우 파일 유형 언급 추가
5. 이제 트리거되어야 하는 5개의 쿼리로 테스트

일반적인 원인:

• 설명이 너무 일반적임
• 일반적인 의역이 누락됨
• 비전문 용어 없이 전문 용어만 사용

수정 템플릿:

# 수정 전 (과소 트리거)
description: 코드 품질 분석

# 수정 후 (특정 트리거)
description: >
  정적 코드 분석 및 품질 평가. 코드 스타일, 복잡성, 보안 취약점,
  테스트 커버리지를 확인합니다. 사용자가 "코드 리뷰", "코드 품질",
  "린트", "정적 분석", "코드 스멜", "코드 감사" 또는 "내 코드 확인"을
  말할 때 사용합니다.

수정: 과다 트리거

1. 현재 설명 읽기
2. 관련 없는 쿼리가 트리거되는 이유 식별
3. 부정 트리거 추가 ("다음에는 사용하지 마세요...")
4. 설명을 더 구체적으로 만들기
5. 트리거되지 않아야 하는 5개의 쿼리로 테스트

수정 템플릿:

# 수정 전 (과다 트리거)
description: 검토를 위해 문서 처리

# 수정 후 (특정 + 부정 트리거)
description: >
  계약 조항 추출 및 규정 준수 검토를 위해 PDF 법률 문서를 처리합니다.
  법률 계약, NDA, 서비스 약관에 사용합니다. 일반 문서 편집, 서식 지정
  또는 비법률 PDF에는 사용하지 마세요.

수정: 실행 문제

1. 워크플로우에서 실패하는 단계 식별
2. 단계 사이에 명시적 검증 게이트 추가
3. 명확한 복구 지침이 포함된 오류 처리 추가
4. "X가 실패하면 Y를 수행" 대체 경로 추가
5. 깨지기 쉬운 작업에 스크립트 추가 고려

수정: 품질 문제

1. 모호한 지침을 구체적인 지침으로 교체
2. 예상 입력/출력의 구체적인 예제 추가
3. 명시적인 "이렇게 하세요, 저렇게 하지 마세요" 비교 추가
4. 최종 출력 전에 품질 검사 단계 추가
5. 검증 스크립트 추가 고려

3단계: 반복 작업 공간 프로토콜

구조화된 작업 공간을 사용하여 반복 전반에 걸쳐 개선 사항 추적:

eval-workspace/
  iteration-1/          # 첫 번째 버전
    eval-0/with_skill/  # 평가 결과
    eval-0/baseline/
    benchmark.json      # 집계된 메트릭
    benchmark.md        # 사람이 읽을 수 있는 보고서
    feedback.json       # 사용자 피드백
  iteration-2/          # 첫 번째 개선 후
    eval-0/with_skill/
    eval-0/baseline/
    benchmark.json
    benchmark.md
    feedback.json

반복 루프:

1. 스킬에 수정 적용
2. /skill-forge eval <path>를 iteration-<N+1>/에서 실행
3. /skill-forge benchmark <path>를 --previous iteration-<N>/와 함께 실행
4. 회귀를 위해 벤치마크 비교 검토
5. 사용자 피드백을 feedback.json에 수집
6. 피드백 읽고 반복 (2단계로 돌아가기)

다음 경우에 반복 중단:

• 사용자가 만족한다고 말함
• 모든 피드백이 비어 있음 (모든 것이 양호함)
• 벤치마크가 반복 간 의미 있는 진전을 보이지 않음
• 통과율이 정의된 임계값을 충족함

3b단계: 자체 어닐링 루프

전체 평가 파이프라인 없이 빠른 수정:

1. 수정 적용
2. 원래 실패 케이스로 테스트
3. 다른 3개 케이스로 테스트 (회귀 확인)
4. 수정이 작동하면:
   -> 지침/SKILL.md 업데이트
   -> 참조 또는 SKILL.md에 학습 내용 문서화
5. 수정이 실패하면:
   -> 이유 진단
   -> 대안 접근 방식 시도
   -> 반복

3c단계: 설명 최적화 루프

트리거 문제(범주 A)의 경우 자동화된 최적화 루프 사용:

1. 트리거 평가 세트 생성: python scripts/generate_eval_set.py <path>
2. 사용자와 평가 세트 검토 및 개선
3. 최적화 실행: python scripts/optimize_description.py <path> --eval-set evals.json
4. 훈련/테스트 분할 점수 및 개선 제안 검토
5. 제안된 설명 변경 적용
6. 개선 측정을 위해 최적화 재실행
7. 가장 높은 테스트 점수를 가진 설명 선택 (훈련 점수 아님 - 과적합 방지)
8. 최대 5회 또는 테스트 점수가 정체될 때까지 반복

4단계: 아키텍처 진화

스킬이 계층을 초과할 때:

계층 1 -> 계층 2 (스크립트 필요):

1. 깨지기 쉬운/결정적 작업 식별
2. scripts/에 스크립트 생성
3. SKILL.md를 업데이트하여 스크립트 참조
4. 스크립트를 독립적으로 테스트

계층 2 -> 계층 3 (하위 스킬 필요):

1. 분리할 수 있는 별개의 워크플로우 식별
2. 각각을 자체 skills/{parent}-{child}/SKILL.md로 추출
3. 라우팅 테이블로 기본 SKILL.md 업데이트
4. 공유 지식을 references/로 이동
5. install.sh 업데이트

계층 3 -> 계층 4 (에이전트 필요):

1. 병렬로 실행할 수 있는 워크플로우 식별
2. agents/에 에이전트 정의 생성
3. 감사/전체 분석 하위 스킬을 업데이트하여 에이전트에 위임
4. 병렬 실행 테스트

5단계: 버전 관리

진화 후:

1. 프론트매터의 metadata.version 업데이트 (있는 경우)
2. SKILL.md 또는 참조 파일에 학습 내용 추가
3. 영향을 받는 모든 상호 참조 업데이트
4. 검증 재실행: python scripts/validate_skill.py <path>
5. 종단 간 전체 워크플로우 테스트

일반적인 진화 패턴

패턴: 업계 탐지 추가

스킬이 사용자 유형에 따라 동작을 조정해야 할 때:

## 업계 탐지
컨텍스트에서 사용자 유형 감지:
- **유형 A**: [신호] -> [동작]
- **유형 B**: [신호] -> [동작]

패턴: 품질 게이트 추가

출력 품질이 일관되지 않을 때:

## 품질 게이트
출력을 전달하기 전에:
- [ ] [확인 1]
- [ ] [확인 2]
- [ ] [확인 3]

패턴: 점수 매기기 추가

사용자가 측정 가능한 출력을 필요로 할 때:

## 점수 매기기 (0-100)
| 범주 | 가중치 |
|----------|--------|
| 범주 A | 30% |
| 범주 B | 30% |
| 범주 C | 20% |
| 범주 D | 20% |

}```yaml
# Before (under-triggers)
description: Analyzes code quality

# After (specific triggers)
description: >
  Static code analysis and quality assessment. Checks code style,
  complexity, security vulnerabilities, and test coverage. Use when
  user says "code review", "code quality", "lint", "static analysis",
  "code smell", "code audit", or "check my code".

Fix: Over-Triggering

1. Read current description
2. Identify why unrelated queries trigger it
3. Add negative triggers ("Do NOT use for...")
4. Make description more specific
5. Test with 5 queries that should NOT trigger

Fix template:

# Before (over-triggers)
description: Processes documents for review

# After (specific + negative triggers)
description: >
  Processes PDF legal documents for contract clause extraction and
  compliance review. Use for legal contracts, NDAs, terms of service.
  Do NOT use for general document editing, formatting, or non-legal PDFs.

Fix: Execution Issues

1. Identify the failing step in the workflow
2. Add explicit validation gates between steps
3. Add error handling with clear recovery instructions
4. Add "If X fails, then Y" fallback paths
5. Consider adding a script for fragile operations

Fix: Quality Issues

1. Replace vague instructions with specific ones
2. Add concrete examples of expected input/output
3. Add explicit "do this, not that" comparisons
4. Add quality check steps before final output
5. Consider adding a validation script

Step 3: Iteration Workspace Protocol

Use structured workspaces to track improvements across iterations:

eval-workspace/
  iteration-1/          # First version
    eval-0/with_skill/  # Eval results
    eval-0/baseline/
    benchmark.json      # Aggregated metrics
    benchmark.md        # Human-readable report
    feedback.json       # User feedback
  iteration-2/          # After first improvement
    eval-0/with_skill/
    eval-0/baseline/
    benchmark.json
    benchmark.md
    feedback.json

The iteration loop:

1. Apply the fix to the skill
2. Run /skill-forge eval <path> into iteration-<N+1>/
3. Run /skill-forge benchmark <path> with --previous iteration-<N>/
4. Review benchmark comparison for regressions
5. Collect user feedback into feedback.json
6. Read feedback and iterate (back to Step 2)

Stop iterating when:

• User says they're happy
• All feedback is empty (everything looks good)
• Benchmark shows no meaningful progress between iterations
• Pass rate meets the defined thresholds

Step 3b: Self-Annealing Loop

For quick fixes without full eval pipeline:

1. Apply the fix
2. Test with the original failing case
3. Test with 3 other cases (regression check)
4. If fix works:
   -> Update the directive/SKILL.md
   -> Document the learning in references or SKILL.md
5. If fix fails:
   -> Diagnose why
   -> Try alternative approach
   -> Repeat

Step 3c: Description Optimization Loop

For triggering issues (Category A), use the automated optimization loop:

1. Generate trigger eval set: python scripts/generate_eval_set.py <path>
2. Review and refine the eval set with the user
3. Run optimization: python scripts/optimize_description.py <path> --eval-set evals.json
4. Review the train/test split scores and improvement suggestions
5. Apply suggested description changes
6. Re-run optimization to measure improvement
7. Select the description with the highest test score (not train — avoids overfitting)
8. Iterate up to 5 times or until test score plateaus

Step 4: Architecture Evolution

When a skill outgrows its tier:

Tier 1 -> Tier 2 (needs scripts):

1. Identify the fragile/deterministic operation
2. Create script in scripts/
3. Update SKILL.md to reference the script
4. Test script independently

Tier 2 -> Tier 3 (needs sub-skills):

1. Identify distinct workflows that can be separated
2. Extract each into its own skills/{parent}-{child}/SKILL.md
3. Update main SKILL.md with routing table
4. Move shared knowledge to references/
5. Update install.sh

Tier 3 -> Tier 4 (needs agents):

1. Identify workflows that can run in parallel
2. Create agent definitions in agents/
3. Update the audit/full-analysis sub-skill to delegate to agents
4. Test parallel execution

Step 5: Version Management

After evolution:

1. Update metadata.version in frontmatter (if present)
2. Add learning to SKILL.md or reference file
3. Update any affected cross-references
4. Re-run validation: python scripts/validate_skill.py <path>
5. Test full workflow end-to-end

Common Evolution Patterns

Pattern: Adding Industry Detection

When a skill needs to adapt behavior by user type:

## Industry Detection
Detect user type from context:
- **Type A**: [signals] -> [behavior]
- **Type B**: [signals] -> [behavior]

Pattern: Adding Quality Gates

When output quality is inconsistent:

## Quality Gates
Before delivering output:
- [ ] [Check 1]
- [ ] [Check 2]
- [ ] [Check 3]

Pattern: Adding Scoring

When users need measurable output:

## Scoring (0-100)
| Category | Weight |
|----------|--------|
| Category A | 30% |
| Category B | 30% |
| Category C | 20% |
| Category D | 20% |

}```yaml
# Before (under-triggers)
description: Analyzes code quality

# After (specific triggers)
description: >
  Static code analysis and quality assessment. Checks code style,
  complexity, security vulnerabilities, and test coverage. Use when
  user says "code review", "code quality", "lint", "static analysis",
  "code smell", "code audit", or "check my code".

Fix: Over-Triggering

1. Read current description
2. Identify why unrelated queries trigger it
3. Add negative triggers ("Do NOT use for...")
4. Make description more specific
5. Test with 5 queries that should NOT trigger

Fix template:

# Before (over-triggers)
description: Processes documents for review

# After (specific + negative triggers)
description: >
  Processes PDF legal documents for contract clause extraction and
  compliance review. Use for legal contracts, NDAs, terms of service.
  Do NOT use for general document editing, formatting, or non-legal PDFs.

Fix: Execution Issues

1. Identify the failing step in the workflow
2. Add explicit validation gates between steps
3. Add error handling with clear recovery instructions
4. Add "If X fails, then Y" fallback paths
5. Consider adding a script for fragile operations

Fix: Quality Issues

1. Replace vague instructions with specific ones
2. Add concrete examples of expected input/output
3. Add explicit "do this, not that" comparisons
4. Add quality check steps before final output
5. Consider adding a validation script

Step 3: Iteration Workspace Protocol

Use structured workspaces to track improvements across iterations:

eval-workspace/
  iteration-1/          # First version
    eval-0/with_skill/  # Eval results
    eval-0/baseline/
    benchmark.json      # Aggregated metrics
    benchmark.md        # Human-readable report
    feedback.json       # User feedback
  iteration-2/          # After first improvement
    eval-0/with_skill/
    eval-0/baseline/
    benchmark.json
    benchmark.md
    feedback.json

The iteration loop:

1. Apply the fix to the skill
2. Run /skill-forge eval <path> into iteration-<N+1>/
3. Run /skill-forge benchmark <path> with --previous iteration-<N>/
4. Review benchmark comparison for regressions
5. Collect user feedback into feedback.json
6. Read feedback and iterate (back to Step 2)

Stop iterating when:

• User says they're happy
• All feedback is empty (everything looks good)
• Benchmark shows no meaningful progress between iterations
• Pass rate meets the defined thresholds

Step 3b: Self-Annealing Loop

For quick fixes without full eval pipeline:

1. Apply the fix
2. Test with the original failing case
3. Test with 3 other cases (regression check)
4. If fix works:
   -> Update the directive/SKILL.md
   -> Document the learning in references or SKILL.md
5. If fix fails:
   -> Diagnose why
   -> Try alternative approach
   -> Repeat

Step 3c: Description Optimization Loop

For triggering issues (Category A), use the automated optimization loop:

1. Generate trigger eval set: python scripts/generate_eval_set.py <path>
2. Review and refine the eval set with the user
3. Run optimization: python scripts/optimize_description.py <path> --eval-set evals.json
4. Review the train/test split scores and improvement suggestions
5. Apply suggested description changes
6. Re-run optimization to measure improvement
7. Select the description with the highest test score (not train — avoids overfitting)
8. Iterate up to 5 times or until test score plateaus

Step 4: Architecture Evolution

When a skill outgrows its tier:

Tier 1 -> Tier 2 (needs scripts):

1. Identify the fragile/deterministic operation
2. Create script in scripts/
3. Update SKILL.md to reference the script
4. Test script independently

Tier 2 -> Tier 3 (needs sub-skills):

1. Identify distinct workflows that can be separated
2. Extract each into its own skills/{parent}-{child}/SKILL.md
3. Update main SKILL.md with routing table
4. Move shared knowledge to references/
5. Update install.sh

Tier 3 -> Tier 4 (needs agents):

1. Identify workflows that can run in parallel
2. Create agent definitions in agents/
3. Update the audit/full-analysis sub-skill to delegate to agents
4. Test parallel execution

Step 5: Version Management

After evolution:

1. Update metadata.version in frontmatter (if present)
2. Add learning to SKILL.md or reference file
3. Update any affected cross-references
4. Re-run validation: python scripts/validate_skill.py <path>
5. Test full workflow end-to-end

Common Evolution Patterns

Pattern: Adding Industry Detection

When a skill needs to adapt behavior by user type:

## Industry Detection
Detect user type from context:
- **Type A**: [signals] -> [behavior]
- **Type B**: [signals] -> [behavior]

Pattern: Adding Quality Gates

When output quality is inconsistent:

## Quality Gates
Before delivering output:
- [ ] [Check 1]
- [ ] [Check 2]
- [ ] [Check 3]

Pattern: Adding Scoring

When users need measurable output:

## Scoring (0-100)
| Category | Weight |
|----------|--------|
| Category A | 30% |
| Category B | 30% |
| Category C | 20% |
| Category D | 20% |

}```yaml
# Before (under-triggers)
description: Analyzes code quality

# After (specific triggers)
description: >
  Static code analysis and quality assessment. Checks code style,
  complexity, security vulnerabilities, and test coverage. Use when
  user says "code review", "code quality", "lint", "static analysis",
  "code smell", "code audit", or "check my code".

Fix: Over-Triggering

1. Read current description
2. Identify why unrelated queries trigger it
3. Add negative triggers ("Do NOT use for...")
4. Make description more specific
5. Test with 5 queries that should NOT trigger

Fix template:

# Before (over-triggers)
description: Processes documents for review

# After (specific + negative triggers)
description: >
  Processes PDF legal documents for contract clause extraction and
  compliance review. Use for legal contracts, NDAs, terms of service.
  Do NOT use for general document editing, formatting, or non-legal PDFs.

Fix: Execution Issues

1. Identify the failing step in the workflow
2. Add explicit validation gates between steps
3. Add error handling with clear recovery instructions
4. Add "If X fails, then Y" fallback paths
5. Consider adding a script for fragile operations

Fix: Quality Issues

1. Replace vague instructions with specific ones
2. Add concrete examples of expected input/output
3. Add explicit "do this, not that" comparisons
4. Add quality check steps before final output
5. Consider adding a validation script

Step 3: Iteration Workspace Protocol

Use structured workspaces to track improvements across iterations:

eval-workspace/
  iteration-1/          # First version
    eval-0/with_skill/  # Eval results
    eval-0/baseline/
    benchmark.json      # Aggregated metrics
    benchmark.md        # Human-readable report
    feedback.json       # User feedback
  iteration-2/          # After first improvement
    eval-0/with_skill/
    eval-0/baseline/
    benchmark.json
    benchmark.md
    feedback.json

The iteration loop:

1. Apply the fix to the skill
2. Run /skill-forge eval <path> into iteration-<N+1>/
3. Run /skill-forge benchmark <path> with --previous iteration-<N>/
4. Review benchmark comparison for regressions
5. Collect user feedback into feedback.json
6. Read feedback and iterate (back to Step 2)

Stop iterating when:

• User says they're happy
• All feedback is empty (everything looks good)
• Benchmark shows no meaningful progress between iterations
• Pass rate meets the defined thresholds

Step 3b: Self-Annealing Loop

For quick fixes without full eval pipeline:

1. Apply the fix
2. Test with the original failing case
3. Test with 3 other cases (regression check)
4. If fix works:
   -> Update the directive/SKILL.md
   -> Document the learning in references or SKILL.md
5. If fix fails:
   -> Diagnose why
   -> Try alternative approach
   -> Repeat

Step 3c: Description Optimization Loop

For triggering issues (Category A), use the automated optimization loop:

1. Generate trigger eval set: python scripts/generate_eval_set.py <path>
2. Review and refine the eval set with the user
3. Run optimization: python scripts/optimize_description.py <path> --eval-set evals.json
4. Review the train/test split scores and improvement suggestions
5. Apply suggested description changes
6. Re-run optimization to measure improvement
7. Select the description with the highest test score (not train — avoids overfitting)
8. Iterate up to 5 times or until test score plateaus

Step 4: Architecture Evolution

When a skill outgrows its tier:

Tier 1 -> Tier 2 (needs scripts):

1. Identify the fragile/deterministic operation
2. Create script in scripts/
3. Update SKILL.md to reference the script
4. Test script independently

Tier 2 -> Tier 3 (needs sub-skills):

1. Identify distinct workflows that can be separated
2. Extract each into its own skills/{parent}-{child}/SKILL.md
3. Update main SKILL.md with routing table
4. Move shared knowledge to references/
5. Update install.sh

Tier 3 -> Tier 4 (needs agents):

1. Identify workflows that can run in parallel
2. Create agent definitions in agents/
3. Update the audit/full-analysis sub-skill to delegate to agents
4. Test parallel execution

Step 5: Version Management

After evolution:

1. Update metadata.version in frontmatter (if present)
2. Add learning to SKILL.md or reference file
3. Update any affected cross-references
4. Re-run validation: python scripts/validate_skill.py <path>
5. Test full workflow end-to-end

Common Evolution Patterns

Pattern: Adding Industry Detection

When a skill needs to adapt behavior by user type:

## Industry Detection
Detect user type from context:
- **Type A**: [signals] -> [behavior]
- **Type B**: [signals] -> [behavior]

Pattern: Adding Quality Gates

When output quality is inconsistent:

## Quality Gates
Before delivering output:
- [ ] [Check 1]
- [ ] [Check 2]
- [ ] [Check 3]

Pattern: Adding Scoring

When users need measurable output:

## Scoring (0-100)
| Category | Weight |
|----------|--------|
| Category A | 30% |
| Category B | 30% |
| Category C | 20% |
| Category D | 20% |

}