Community写作与编辑github.com

CoderMoray/HaluCatch

AI Skill 执行可靠性审查工具。评估 Skill 被 AI 执行时的可复现性、可信度与业务适配性。Halu = Hallucination, Catch = 捕获。

HaluCatch 是什么?

HaluCatch is a Claude Code agent skill that aI Skill 执行可靠性审查工具。评估 Skill 被 AI 执行时的可复现性、可信度与业务适配性。Halu = Hallucination, Catch = 捕获。.

兼容平台✓Claude Code~Codex CLI✓Cursor
npx skills add CoderMoray/HaluCatch

Installed? Explore more 写作与编辑 skills: steipete/notion, langchain-ai/langchain, bytedance/podcast-generation · View all 6 →

在你喜欢的 AI 中提问

打开一个已预加载此 Agent Skill 的新对话。

文档

HaluCatch — AI Skill Execution Reliability Auditor

Evaluates the reliability of a Skill package when executed by an AI, producing an evaluation report and fix recommendations.


Capability Boundaries

Good AtNot Good At
Assessing whether AI execution of a skill will produce errorsCybersecurity audits (SQL injection, XSS, etc.)
Checking if data pipelines are reliableCompliance audits (GDPR, privacy regulations, etc.)
Detecting ambiguity in natural language business rulesSkill's own business correctness (does not understand business logic)
Checking if interpretation guardrails are in placeCode performance optimization
Outputting fix recommendations and skeleton scriptsReplacing human business decisions

Role

When the user invokes HaluCatch, you are the HaluCatch audit executor, not a bystander.

Responsibilities

  • Read all files in the target Skill directory
  • Execute the evaluation following the four-dimensional framework
  • Call halucatch_core.py to complete the full workflow in one go (L1 scan + L2 evaluation + L3 report generation)
  • Read the report files generated by the script, present the standard version in the conversation, and ask if the user wants to fix issues
  • Supplement semantic analysis (info-level items) on top of the script's report

Division of Labor Between You and halucatch_core.py

LevelTaskExecutorPrinciple
L1File scanninghalucatch_core.py --validateHigh determinism; script is faster and more accurate
L2Foundation + Code + Rules + Guardrails checksScript reads JSON baseline → you supplement analysis on top of itRegex matches by script, context interpretation by you
L3Three-version report generationhalucatch_core.py generates and writes files, you read and present to userreporter.py is high-determinism, consistent format, zero-hallucination—you only do semantic supplementation

Core Principle: halucatch_core.py covers the full workflow—L1 scan, L2 evaluation, L3 report generation all in one shot. You only need to read the generated report and present it to the user. AI no longer independently writes the report body.

Permissions and Security Boundaries

HaluCatch only needs the following permissions to run:

OperationRequiredDescription
Read target Skill directoryReadRecursively read all files
Write report filesWriteWrite only to reports/ directory, do not modify the target Skill
Execute Python scriptsBashExecute only halucatch_core.py, no network requests

Security Constraints: HaluCatch will not access any path outside the target directory, will not initiate network connections, and will not modify the audited Skill files. The target path must be explicitly specified by the user before auditing.


Input

The user provides a Skill directory path. This directory may contain:

File TypeRequiredDescription
SKILL.md✅Main instruction file for the Skill
manifest.json or config.yaml❌Skill configuration file (includes version number, etc.)
*.py❌Canned scripts for data pipelines (if any)
*.xlsx / *.csv etc.❌Data files (if any, used for reconciliation validation)

Data Requirements

  • Recency: The audit is based on an immediate snapshot of the target Skill directory; no historical versions are traced.
  • Data Scope: Only evaluates files visible in the target directory; does not crawl external dependencies or network resources.

Assumptions

  • The target directory exists and has read permissions.
  • The target Skill should include at least one SKILL.md file (canonical name). If other .md files are present, the AI will attempt heuristic matching but will report naming issues.
  • .py files in the directory are considered the Skill's core execution scripts, not third-party dependencies.

Example User Queries

User SaysAI Action
"Please use HaluCatch to audit this Skill: /path/to/skill"Standard workflow: scan → classify → four-dimensional evaluation → three-version output
"Audit ./data-skill, focus on the data pipeline"Classify as code-engineering type, emphasize Foundation and Code dimensions
"Audit ./guide-skill, check if instructions are clear enough"Classify as pure methodology type, output methodology evaluation
"Scan ./my-skill with --validate mode"Output only file inventory and type classification, no evaluation
"Audit this Skill and fix any issues found"Evaluate → user confirms "fix" → automatically generate fix plan

AI Execution Guide

Automatic Language Detection

When loading this Skill, the AI should detect the user's language from the system prompt (<response_language>) or conversation context, and automatically add the --lang parameter:

User LanguageParameterExample
Chinese (Simplified/Traditional)--lang zh-CNpython3 halucatch_core.py --skill-dir <path> --lang zh-CN
English--lang enpython3 halucatch_core.py --skill-dir <path> --lang en
UncertainDo not add (default auto, auto-detect system locale)python3 halucatch_core.py --skill-dir <path>

Principle: The AI definitely knows what language the user is speaking; no manual configuration needed.

Basic Usage

# Audit for a Chinese-speaking user
python3 halucatch_core.py --skill-dir /path/to/skill --lang zh-CN

# Audit for an English-speaking user
python3 halucatch_core.py --skill-dir /path/to/skill --lang en

# Auto-detect (fallback)
python3 halucatch_core.py --skill-dir /path/to/skill

Execution Flow

Execution Paradigm: Each Phase allocates responsibilities according to the three-tier invocation model. halucatch_core.py completes L1/L2/L3 in one go; you read the generated report files and present them to the user.

Phase 0: Skill Classification

First, determine the type of this Skill:

Does this Skill involve data processing?
  ├─ ❌ Pure Methodology type (instructions/templates/documentation)
  │    Evaluation focus: instruction completeness, logical coherence, reproducibility
  │
  ├─ ✅ Code-Engineering type (contains .py / data files / inline code in md)
  │    Evaluation focus: Foundation + Code + Rules + Guardrails
  │
  └─ ⚠️ Uncertain
        → Ask user: "Does this Skill have data processing steps?"

If the directory contains .xlsx, .csv, .py (including pd.read_, pd.DataFrame) files or content, default to "code-engineering type."

Phase 1: File Scanning

Read all files in the directory and build an inventory:

Information PointOutput Format
File name listTable (File Name / Size / Type)
SKILL.md total linesNumber
.py file line count (if any)Number
Data files listTable
Skill name and descriptionExtracted from frontmatter

Phase 2: Multi-Dimensional Evaluation

Based on the classification from Phase 0, execute checks for the corresponding evaluation dimensions.


2a. Code-Engineering Skill Evaluation

🏗️ Foundation Evaluation

Check whether the Skill's data pipeline is solid. The weaker the foundation, the higher the probability of AI errors when writing code autonomously.

Check ItemPass Criteria
Has canned .py scriptScript exists in the folder
Path parameterizedNumber of hardcoded paths = 0
Column pre-check / input validationHas check_columns or similar function
validate modeHas --validate or similar mode
File discovery mechanismUses glob/wildcards rather than fixed filenames
Dependency declarationDeclares required Python packages in SKILL.md
skiprows parameterizedExcel read row count configurable or auto-detected

Rating: 🟢 Solid / 🟡 Has issues / 🔴 No foundation


🤖 Code Risk Evaluation

If the SKILL.md contains embedded Python code (which the AI must reproduce verbatim), check for these tampering points:

Check CategoryHigh-Risk PatternAI's Potential Erroneous Change
Statistical functionsCustom p-value formula, Z-score calculationReplaced with scipy.stats / constant adjusted
String matchingclean_rate() parsing percentageReplaced replace('%','') with a different implementation
Float comparison== 0 / == 1Changed to math.isclose() (semantically different)
Exception handlingBare except: passChanged to specific exception types
Conditional logic(condition).sum() > 0Changed to .any(axis=1) (different behavior)
Aggregation logicunstack(fill_value=0)Removed fill_value (NaN propagation)
Data cleaningNo data type conversion instructionsMay attempt sum() on string column and error

Rating: 🟢 Low risk / 🟠 Risky / 🔴 High risk


📝 Rules and Criteria Evaluation

Check whether business rules are described clearly and unambiguously in SKILL.md:

Check ItemRisk
Channel/category criteria explicitly listedAmbiguity → AI guesses
Edge case / boundary conditions definedMissed → AI handles arbitrarily
Ratio/mapping relationships canned (not relying on AI knowledge)Not canned → different AI produces different results
Same-store, year-over-year comparison criteria definedNot defined → AI decides arbitrarily, not auditable
Special correction rules (e.g., HaiNanJuShi) documentedNot documented → critical business logic missed
Filename/column name/data format conventions agreedNot agreed → pipeline fails to run

Rating: 🟢 Clear / 🟡 Ambiguous / 🔴 Major omissions


🛡️ Interpretation Guardrails Evaluation

Check whether SKILL.md constrains how the AI interprets the results:

Check ItemSeverity
Causal language prohibitionMissing → 🔴 Critical
Four-quadrant / effect size frameworkMissing → 🟠 High
Self-check mechanismMissing → 🟠 High
Multiple comparison reminderMissing → 🟠 High
Limitation statement templateMissing → 🟠 High
Staged state outputMissing → 🟡 Medium
Data summary statistics printMissing → 🟡 Medium
Output format definitionMissing → 🟡 Medium

Rating: 🟢 Comprehensive / 🟡 Missing items / 🔴 No guardrails


2b. Pure Methodology Skill Evaluation

Check ItemDescription
Instruction completenessEvery step has clear input/output/decision conditions
Edge casesHas "if... then..." exception branch handling
ReproducibilityWill different AI executions reach consistent conclusions?
Example-drivenContains concrete examples to illustrate expected output
Output format definitionAI output structure is constrained
Self-verificationCan the Skill's execution results be cross-checked?

Rating: 🟢 Reliable / 🟡 Room for improvement / 🔴 Unreliable


⚠️ Confirm Before Execution: Before starting file scanning, running halucatch_core.py, or generating reports, you must first confirm with the user that the target path is correct and inform them of the upcoming operations (reading target directory files, running a local script, generating reports to the reports/ directory). Do not perform any read or write operations before explicit confirmation.

Phase 3: Three-Version Output

Core Change: Reports are automatically generated by halucatch_core.py (reporter.py). You no longer need to independently write the report body. You are only responsible for:

  1. Reading the report files generated by the script
  2. Presenting the standard version in the conversation
  3. Supplementing semantic analysis (associating info-level items) on top of the generated report

Output Decision Rules: The three-version report is always generated by the script and written to disk; only the standard version is shown in the conversation.

  1. Run halucatch_core.py --skill-dir <path> → Script automatically completes L1 scan + L2 evaluation + L3 report generation
  2. Three .md files are written to the HaluCatch/reports/ directory (or the path specified by --output-dir)
  3. Only the standard version (plain language, zero jargon) is output in the conversation
  4. The standard version ends with a prompt: "Need technical details (professional version) or a fix plan (action version)?"
  5. If the user says "Yes" → Show the corresponding version content in the conversation (the file is already on disk, just read and display it)
  6. If the user does not respond → Do not proactively output; avoid information overload

1. Professional Version (for Data Analysts / Engineers)

Automatically generated by reporter.py, includes TL;DR summary, four-dimensional rating matrix table, dimension-by-dimension findings list, and check statement.

2. Standard Version (for Business Stakeholders / Non-Technical Users)

Automatically generated by reporter.py, includes plain language summary, 21 context interpretation mappings, zero-jargon output.

3. AI Action Version (for Fix Phase Use)

Automatically generated by reporter.py, includes fix checklist, verification checkpoints, and three-option step prompt.

Report Check Statement

All three report versions must end with a check line:

> This report was generated by HaluCatch. Check status: [✅ All four dimensions of HaluCatch evaluation completed / ⚠️ Some evaluation dimensions incomplete].

If any dimension was not covered during evaluation (e.g., a code-engineering Skill but the user did not provide .py files), the check status downgrades to ⚠️.

AI Semantic Supplementation (on top of script report)

After reading the script-generated report, review each finding item-by-item and perform context analysis, ordered by severity from high to low:

Original RatingAI Needs to Judge
🔴 BlockingHow severe is this risk in the actual business context? Will it really cause execution failure? Fix priority?
🟠 High RiskDoes the context actually constitute a risk? Could the script have flagged a false positive? Does the fix plan need refinement?
🟡 InfoWas the skipped item actually justified? Are there important risks the script missed?
🟢 Pass✅ Script judgment is correct; no supplementation needed

Output format: Append one line > **AI Analysis**: [Specific judgment] below each finding item.


Phase 4: Fix Decision & Closure

After the evaluation is complete, present the standard version report to the user and ask:

Detected [N] risks. Fix according to the recommended plan?

  • User says "Fix" → Generate a fix plan, then present a three-option choice:

    Fix plan generated. Please choose:

    1. Execute Fix — Send the fix plan to your AI to modify the target Skill according to the plan
    2. Do Not Execute — Do not make any changes, end this audit
    3. I Have a Better Idea — Describe your idea, and I will regenerate the fix plan accordingly
    • User selects "Execute" → Prompt the user to have the AI apply the fix → Prompt to re-run HaluCatch for verification after fixing
    • User selects "Do Not Execute" → End
    • User selects "Better Idea" → Re-analyze with additional requirements → Return to "Generate fix plan"
  • User says "Don't fix" → End


Report Persistence

  • Default output to the HaluCatch/reports/ directory (does not pollute the target Skill directory)
  • Use --output-dir to output to a custom path
  • Fix package (if any) saved to: {SkillDirectory}/halucatch-fix/

More Trigger Examples

By Audit Depth

User SaysAI Action
"Audit this Skill for me, see if it's reliable"Full workflow: classify → four-dimensional evaluation → three-version report
"Quick scan, any obvious pitfalls?"Only Phase 1 scan + Phase 2 L1/L2 rule checks, output a concise checklist
"This time only focus on hard issues like div by zero or bare except in the code"Skip Rules and Guardrails dimensions; only run Foundation + Code checks
"I modified SKILL.md after the last audit, run it again and compare"Re-audit, compare with the previous report, annotate fix status

By Skill Type

User SaysAI Action
"My Skill has a data/ folder and .py scripts, audit it comprehensively"Classify as code-engineering type, full four dimensions
"This is a pure guidance document skill, check if the instructions are clear"Classify as pure methodology type, evaluate only methodology and guardrails

Path Variations (Platform Differences)

PlatformCorrect Format❌ Incorrect Format
Claude Code / General/path/to/skillC:\path\to\skill
Kimi / WeChat Mini Programskills/project-name/~/skills/project-name/
Cursor / VS CodeDrag and drop the folder into the dialogManually typing complex paths

Still Stuck?

Just say: "Audit this Skill for me" and drag in the Skill folder, or paste a portion of the SKILL.md content. The AI will determine the next steps.


Error Handling

Common Errors and Fixes

Error SymptomCauseFix Method
❌ Cannot find SKILL.mdTarget directory does not exist or has no .md filesConfirm the path is correct → Check if the directory contains SKILL.md or other .md files → If not, create one
⚠️ No standard SKILL.md foundFilename is not SKILL.md (e.g., skill.md, README.md)Rename the file to SKILL.md, or tell the AI to use --file to specify the filename
🔧 File encoding error, skipped XXX.pyFile contains non-UTF-8 characters (e.g., GBK-encoded Chinese comments)Use an editor to save the file as UTF-8, or change comments to English
📦 Skill package too large (> 2MB)Directory contains large data files or dependency packagesKeep only core files (SKILL.md + .py scripts); place data files and dependencies in .halucatch-ignore
⏱️ Audit timeoutToo many files or script execution takes too longTell the AI: "Only audit SKILL.md and core .py files"

Error Severity Levels

LevelBehavior
Fatal (e.g., directory does not exist)Terminate immediately, provide clear fix instructions
Warning (e.g., non-standard filename)Continue audit but downgrade self-check score, annotate in report
Recoverable (e.g., single file encoding issue)Skip that file and continue, annotate skipped files and reason in report

Operational Stability

Protection Measures

ScenarioProtection Strategy
Large file (single > 1MB)Analyze only the first 500 lines, declare truncation in report
Too many files (directory > 200 files)Filter by type (.md → .py → other); non-core files skipped automatically
Script execution timeout (> 30s)Terminate that step, write verified parts into report, annotate incomplete items
Network requestsHaluCatch does not make network requests, 100% offline
Encoding issuesTry UTF-8 first → then system locale → skip and log on failure
Directory unreadableReport permission error, suggest user chmod or try another path

Reliability Statement

HaluCatch runs stably on normal Skill directories (≤ 50 files, single file ≤ 1MB). Extreme cases will automatically degrade gracefully (truncate/skip/terminate) without silent failure.


Anti-Patterns & FAQ

Common Misconceptions

MisconceptionCorrect Understanding
"One audit is enough"The Skill should be re-audited after every modification, especially SKILL.md or key .py files
"Low score = unusable"Score is a relative reference. A "weak foundation but clear rules" pure methodology Skill might be perfectly usable
"Must fix all issues before release"Prioritize high-severity (🔴) and medium-severity (🟠) items; low-severity items can be improved incrementally
"The AI Action version report can be executed directly"The action version is a fix instruction for the AI; user confirmation is required before the AI executes, to prevent accidental modification
"Running --validate mode counts as auditing"--validate only does file scanning and type classification, not the four-dimensional evaluation. A formal audit requires the full run

FAQ

Q: What do I need to prepare? A: A folder containing SKILL.md. If there are .py scripts or data files, include them for a more comprehensive evaluation.

Q: The audit result says it doesn't pass. What should I do? A: Look at the "AI Action Version" in the report, which contains item-by-item fix plans. Fix in order of priority from high to low, then re-audit to verify.

Q: My Skill has no Python code. Can I use this? A: Yes. HaluCatch will automatically classify it as "pure methodology type," skip Foundation and Code checks, and focus on instruction completeness and guardrails.

Q: What if I get an error? A: See the "Error Handling" section above. 90% of errors are due to incorrect paths or non-standard filenames. If you can't resolve it, paste the error message to the AI.

💡 More questions? See FAQ.md — contains a complete usage guide, common questions, and troubleshooting.

相关技能