HaluCatch — AI Skill Execution Reliability Auditor
Evaluates the reliability of a Skill package when executed by an AI, producing an evaluation report and fix recommendations.
Capability Boundaries
| Good At | Not Good At |
|---|---|
| Assessing whether AI execution of a skill will produce errors | Cybersecurity audits (SQL injection, XSS, etc.) |
| Checking if data pipelines are reliable | Compliance audits (GDPR, privacy regulations, etc.) |
| Detecting ambiguity in natural language business rules | Skill's own business correctness (does not understand business logic) |
| Checking if interpretation guardrails are in place | Code performance optimization |
| Outputting fix recommendations and skeleton scripts | Replacing human business decisions |
Role
When the user invokes HaluCatch, you are the HaluCatch audit executor, not a bystander.
Responsibilities
- Read all files in the target Skill directory
- Execute the evaluation following the four-dimensional framework
- Call
halucatch_core.pyto complete the full workflow in one go (L1 scan + L2 evaluation + L3 report generation) - Read the report files generated by the script, present the standard version in the conversation, and ask if the user wants to fix issues
- Supplement semantic analysis (
info-level items) on top of the script's report
Division of Labor Between You and halucatch_core.py
| Level | Task | Executor | Principle |
|---|---|---|---|
| L1 | File scanning | halucatch_core.py --validate | High determinism; script is faster and more accurate |
| L2 | Foundation + Code + Rules + Guardrails checks | Script reads JSON baseline → you supplement analysis on top of it | Regex matches by script, context interpretation by you |
| L3 | Three-version report generation | halucatch_core.py generates and writes files, you read and present to user | reporter.py is high-determinism, consistent format, zero-hallucination—you only do semantic supplementation |
Core Principle:
halucatch_core.pycovers the full workflow—L1 scan, L2 evaluation, L3 report generation all in one shot. You only need to read the generated report and present it to the user. AI no longer independently writes the report body.
Permissions and Security Boundaries
HaluCatch only needs the following permissions to run:
| Operation | Required | Description |
|---|---|---|
| Read target Skill directory | Read | Recursively read all files |
| Write report files | Write | Write only to reports/ directory, do not modify the target Skill |
| Execute Python scripts | Bash | Execute only halucatch_core.py, no network requests |
Security Constraints: HaluCatch will not access any path outside the target directory, will not initiate network connections, and will not modify the audited Skill files. The target path must be explicitly specified by the user before auditing.
Input
The user provides a Skill directory path. This directory may contain:
| File Type | Required | Description |
|---|---|---|
SKILL.md | ✅ | Main instruction file for the Skill |
manifest.json or config.yaml | ❌ | Skill configuration file (includes version number, etc.) |
*.py | ❌ | Canned scripts for data pipelines (if any) |
*.xlsx / *.csv etc. | ❌ | Data files (if any, used for reconciliation validation) |
Data Requirements
- Recency: The audit is based on an immediate snapshot of the target Skill directory; no historical versions are traced.
- Data Scope: Only evaluates files visible in the target directory; does not crawl external dependencies or network resources.
Assumptions
- The target directory exists and has read permissions.
- The target Skill should include at least one
SKILL.mdfile (canonical name). If other.mdfiles are present, the AI will attempt heuristic matching but will report naming issues. .pyfiles in the directory are considered the Skill's core execution scripts, not third-party dependencies.
Example User Queries
| User Says | AI Action |
|---|---|
| "Please use HaluCatch to audit this Skill: /path/to/skill" | Standard workflow: scan → classify → four-dimensional evaluation → three-version output |
| "Audit ./data-skill, focus on the data pipeline" | Classify as code-engineering type, emphasize Foundation and Code dimensions |
| "Audit ./guide-skill, check if instructions are clear enough" | Classify as pure methodology type, output methodology evaluation |
| "Scan ./my-skill with --validate mode" | Output only file inventory and type classification, no evaluation |
| "Audit this Skill and fix any issues found" | Evaluate → user confirms "fix" → automatically generate fix plan |
AI Execution Guide
Automatic Language Detection
When loading this Skill, the AI should detect the user's language from the system prompt (<response_language>) or conversation context, and automatically add the --lang parameter:
| User Language | Parameter | Example |
|---|---|---|
| Chinese (Simplified/Traditional) | --lang zh-CN | python3 halucatch_core.py --skill-dir <path> --lang zh-CN |
| English | --lang en | python3 halucatch_core.py --skill-dir <path> --lang en |
| Uncertain | Do not add (default auto, auto-detect system locale) | python3 halucatch_core.py --skill-dir <path> |
Principle: The AI definitely knows what language the user is speaking; no manual configuration needed.
Basic Usage
# Audit for a Chinese-speaking user
python3 halucatch_core.py --skill-dir /path/to/skill --lang zh-CN
# Audit for an English-speaking user
python3 halucatch_core.py --skill-dir /path/to/skill --lang en
# Auto-detect (fallback)
python3 halucatch_core.py --skill-dir /path/to/skill
Execution Flow
Execution Paradigm: Each Phase allocates responsibilities according to the three-tier invocation model.
halucatch_core.pycompletes L1/L2/L3 in one go; you read the generated report files and present them to the user.
Phase 0: Skill Classification
First, determine the type of this Skill:
Does this Skill involve data processing?
├─ ❌ Pure Methodology type (instructions/templates/documentation)
│ Evaluation focus: instruction completeness, logical coherence, reproducibility
│
├─ ✅ Code-Engineering type (contains .py / data files / inline code in md)
│ Evaluation focus: Foundation + Code + Rules + Guardrails
│
└─ ⚠️ Uncertain
→ Ask user: "Does this Skill have data processing steps?"
If the directory contains .xlsx, .csv, .py (including pd.read_, pd.DataFrame) files or content, default to "code-engineering type."
Phase 1: File Scanning
Read all files in the directory and build an inventory:
| Information Point | Output Format |
|---|---|
| File name list | Table (File Name / Size / Type) |
| SKILL.md total lines | Number |
| .py file line count (if any) | Number |
| Data files list | Table |
| Skill name and description | Extracted from frontmatter |
Phase 2: Multi-Dimensional Evaluation
Based on the classification from Phase 0, execute checks for the corresponding evaluation dimensions.
2a. Code-Engineering Skill Evaluation
🏗️ Foundation Evaluation
Check whether the Skill's data pipeline is solid. The weaker the foundation, the higher the probability of AI errors when writing code autonomously.
| Check Item | Pass Criteria |
|---|---|
| Has canned .py script | Script exists in the folder |
| Path parameterized | Number of hardcoded paths = 0 |
| Column pre-check / input validation | Has check_columns or similar function |
| validate mode | Has --validate or similar mode |
| File discovery mechanism | Uses glob/wildcards rather than fixed filenames |
| Dependency declaration | Declares required Python packages in SKILL.md |
| skiprows parameterized | Excel read row count configurable or auto-detected |
Rating: 🟢 Solid / 🟡 Has issues / 🔴 No foundation
🤖 Code Risk Evaluation
If the SKILL.md contains embedded Python code (which the AI must reproduce verbatim), check for these tampering points:
| Check Category | High-Risk Pattern | AI's Potential Erroneous Change |
|---|---|---|
| Statistical functions | Custom p-value formula, Z-score calculation | Replaced with scipy.stats / constant adjusted |
| String matching | clean_rate() parsing percentage | Replaced replace('%','') with a different implementation |
| Float comparison | == 0 / == 1 | Changed to math.isclose() (semantically different) |
| Exception handling | Bare except: pass | Changed to specific exception types |
| Conditional logic | (condition).sum() > 0 | Changed to .any(axis=1) (different behavior) |
| Aggregation logic | unstack(fill_value=0) | Removed fill_value (NaN propagation) |
| Data cleaning | No data type conversion instructions | May attempt sum() on string column and error |
Rating: 🟢 Low risk / 🟠 Risky / 🔴 High risk
📝 Rules and Criteria Evaluation
Check whether business rules are described clearly and unambiguously in SKILL.md:
| Check Item | Risk |
|---|---|
| Channel/category criteria explicitly listed | Ambiguity → AI guesses |
| Edge case / boundary conditions defined | Missed → AI handles arbitrarily |
| Ratio/mapping relationships canned (not relying on AI knowledge) | Not canned → different AI produces different results |
| Same-store, year-over-year comparison criteria defined | Not defined → AI decides arbitrarily, not auditable |
| Special correction rules (e.g., HaiNanJuShi) documented | Not documented → critical business logic missed |
| Filename/column name/data format conventions agreed | Not agreed → pipeline fails to run |
Rating: 🟢 Clear / 🟡 Ambiguous / 🔴 Major omissions
🛡️ Interpretation Guardrails Evaluation
Check whether SKILL.md constrains how the AI interprets the results:
| Check Item | Severity |
|---|---|
| Causal language prohibition | Missing → 🔴 Critical |
| Four-quadrant / effect size framework | Missing → 🟠 High |
| Self-check mechanism | Missing → 🟠 High |
| Multiple comparison reminder | Missing → 🟠 High |
| Limitation statement template | Missing → 🟠 High |
| Staged state output | Missing → 🟡 Medium |
| Data summary statistics print | Missing → 🟡 Medium |
| Output format definition | Missing → 🟡 Medium |
Rating: 🟢 Comprehensive / 🟡 Missing items / 🔴 No guardrails
2b. Pure Methodology Skill Evaluation
| Check Item | Description |
|---|---|
| Instruction completeness | Every step has clear input/output/decision conditions |
| Edge cases | Has "if... then..." exception branch handling |
| Reproducibility | Will different AI executions reach consistent conclusions? |
| Example-driven | Contains concrete examples to illustrate expected output |
| Output format definition | AI output structure is constrained |
| Self-verification | Can the Skill's execution results be cross-checked? |
Rating: 🟢 Reliable / 🟡 Room for improvement / 🔴 Unreliable
⚠️ Confirm Before Execution: Before starting file scanning, running
halucatch_core.py, or generating reports, you must first confirm with the user that the target path is correct and inform them of the upcoming operations (reading target directory files, running a local script, generating reports to thereports/directory). Do not perform any read or write operations before explicit confirmation.
Phase 3: Three-Version Output
Core Change: Reports are automatically generated by halucatch_core.py (reporter.py). You no longer need to independently write the report body. You are only responsible for:
- Reading the report files generated by the script
- Presenting the standard version in the conversation
- Supplementing semantic analysis (associating
info-level items) on top of the generated report
Output Decision Rules: The three-version report is always generated by the script and written to disk; only the standard version is shown in the conversation.
- Run
halucatch_core.py --skill-dir <path>→ Script automatically completes L1 scan + L2 evaluation + L3 report generation - Three
.mdfiles are written to theHaluCatch/reports/directory (or the path specified by--output-dir) - Only the standard version (plain language, zero jargon) is output in the conversation
- The standard version ends with a prompt: "Need technical details (professional version) or a fix plan (action version)?"
- If the user says "Yes" → Show the corresponding version content in the conversation (the file is already on disk, just read and display it)
- If the user does not respond → Do not proactively output; avoid information overload
1. Professional Version (for Data Analysts / Engineers)
Automatically generated by reporter.py, includes TL;DR summary, four-dimensional rating matrix table, dimension-by-dimension findings list, and check statement.
2. Standard Version (for Business Stakeholders / Non-Technical Users)
Automatically generated by reporter.py, includes plain language summary, 21 context interpretation mappings, zero-jargon output.
3. AI Action Version (for Fix Phase Use)
Automatically generated by reporter.py, includes fix checklist, verification checkpoints, and three-option step prompt.
Report Check Statement
All three report versions must end with a check line:
> This report was generated by HaluCatch. Check status: [✅ All four dimensions of HaluCatch evaluation completed / ⚠️ Some evaluation dimensions incomplete].
If any dimension was not covered during evaluation (e.g., a code-engineering Skill but the user did not provide .py files), the check status downgrades to ⚠️.
AI Semantic Supplementation (on top of script report)
After reading the script-generated report, review each finding item-by-item and perform context analysis, ordered by severity from high to low:
| Original Rating | AI Needs to Judge |
|---|---|
| 🔴 Blocking | How severe is this risk in the actual business context? Will it really cause execution failure? Fix priority? |
| 🟠 High Risk | Does the context actually constitute a risk? Could the script have flagged a false positive? Does the fix plan need refinement? |
| 🟡 Info | Was the skipped item actually justified? Are there important risks the script missed? |
| 🟢 Pass | ✅ Script judgment is correct; no supplementation needed |
Output format: Append one line > **AI Analysis**: [Specific judgment] below each finding item.
Phase 4: Fix Decision & Closure
After the evaluation is complete, present the standard version report to the user and ask:
Detected [N] risks. Fix according to the recommended plan?
-
User says "Fix" → Generate a fix plan, then present a three-option choice:
Fix plan generated. Please choose:
- Execute Fix — Send the fix plan to your AI to modify the target Skill according to the plan
- Do Not Execute — Do not make any changes, end this audit
- I Have a Better Idea — Describe your idea, and I will regenerate the fix plan accordingly
- User selects "Execute" → Prompt the user to have the AI apply the fix → Prompt to re-run HaluCatch for verification after fixing
- User selects "Do Not Execute" → End
- User selects "Better Idea" → Re-analyze with additional requirements → Return to "Generate fix plan"
-
User says "Don't fix" → End
Report Persistence
- Default output to the
HaluCatch/reports/directory (does not pollute the target Skill directory) - Use
--output-dirto output to a custom path - Fix package (if any) saved to:
{SkillDirectory}/halucatch-fix/
More Trigger Examples
By Audit Depth
| User Says | AI Action |
|---|---|
| "Audit this Skill for me, see if it's reliable" | Full workflow: classify → four-dimensional evaluation → three-version report |
| "Quick scan, any obvious pitfalls?" | Only Phase 1 scan + Phase 2 L1/L2 rule checks, output a concise checklist |
| "This time only focus on hard issues like div by zero or bare except in the code" | Skip Rules and Guardrails dimensions; only run Foundation + Code checks |
| "I modified SKILL.md after the last audit, run it again and compare" | Re-audit, compare with the previous report, annotate fix status |
By Skill Type
| User Says | AI Action |
|---|---|
| "My Skill has a data/ folder and .py scripts, audit it comprehensively" | Classify as code-engineering type, full four dimensions |
| "This is a pure guidance document skill, check if the instructions are clear" | Classify as pure methodology type, evaluate only methodology and guardrails |
Path Variations (Platform Differences)
| Platform | Correct Format | ❌ Incorrect Format |
|---|---|---|
| Claude Code / General | /path/to/skill | C:\path\to\skill |
| Kimi / WeChat Mini Program | skills/project-name/ | ~/skills/project-name/ |
| Cursor / VS Code | Drag and drop the folder into the dialog | Manually typing complex paths |
Still Stuck?
Just say: "Audit this Skill for me" and drag in the Skill folder, or paste a portion of the SKILL.md content. The AI will determine the next steps.
Error Handling
Common Errors and Fixes
| Error Symptom | Cause | Fix Method |
|---|---|---|
❌ Cannot find SKILL.md | Target directory does not exist or has no .md files | Confirm the path is correct → Check if the directory contains SKILL.md or other .md files → If not, create one |
⚠️ No standard SKILL.md found | Filename is not SKILL.md (e.g., skill.md, README.md) | Rename the file to SKILL.md, or tell the AI to use --file to specify the filename |
🔧 File encoding error, skipped XXX.py | File contains non-UTF-8 characters (e.g., GBK-encoded Chinese comments) | Use an editor to save the file as UTF-8, or change comments to English |
📦 Skill package too large (> 2MB) | Directory contains large data files or dependency packages | Keep only core files (SKILL.md + .py scripts); place data files and dependencies in .halucatch-ignore |
⏱️ Audit timeout | Too many files or script execution takes too long | Tell the AI: "Only audit SKILL.md and core .py files" |
Error Severity Levels
| Level | Behavior |
|---|---|
| Fatal (e.g., directory does not exist) | Terminate immediately, provide clear fix instructions |
| Warning (e.g., non-standard filename) | Continue audit but downgrade self-check score, annotate in report |
| Recoverable (e.g., single file encoding issue) | Skip that file and continue, annotate skipped files and reason in report |
Operational Stability
Protection Measures
| Scenario | Protection Strategy |
|---|---|
| Large file (single > 1MB) | Analyze only the first 500 lines, declare truncation in report |
| Too many files (directory > 200 files) | Filter by type (.md → .py → other); non-core files skipped automatically |
| Script execution timeout (> 30s) | Terminate that step, write verified parts into report, annotate incomplete items |
| Network requests | HaluCatch does not make network requests, 100% offline |
| Encoding issues | Try UTF-8 first → then system locale → skip and log on failure |
| Directory unreadable | Report permission error, suggest user chmod or try another path |
Reliability Statement
HaluCatch runs stably on normal Skill directories (≤ 50 files, single file ≤ 1MB). Extreme cases will automatically degrade gracefully (truncate/skip/terminate) without silent failure.
Anti-Patterns & FAQ
Common Misconceptions
| Misconception | Correct Understanding |
|---|---|
| "One audit is enough" | The Skill should be re-audited after every modification, especially SKILL.md or key .py files |
| "Low score = unusable" | Score is a relative reference. A "weak foundation but clear rules" pure methodology Skill might be perfectly usable |
| "Must fix all issues before release" | Prioritize high-severity (🔴) and medium-severity (🟠) items; low-severity items can be improved incrementally |
| "The AI Action version report can be executed directly" | The action version is a fix instruction for the AI; user confirmation is required before the AI executes, to prevent accidental modification |
"Running --validate mode counts as auditing" | --validate only does file scanning and type classification, not the four-dimensional evaluation. A formal audit requires the full run |
FAQ
Q: What do I need to prepare?
A: A folder containing SKILL.md. If there are .py scripts or data files, include them for a more comprehensive evaluation.
Q: The audit result says it doesn't pass. What should I do? A: Look at the "AI Action Version" in the report, which contains item-by-item fix plans. Fix in order of priority from high to low, then re-audit to verify.
Q: My Skill has no Python code. Can I use this? A: Yes. HaluCatch will automatically classify it as "pure methodology type," skip Foundation and Code checks, and focus on instruction completeness and guardrails.
Q: What if I get an error? A: See the "Error Handling" section above. 90% of errors are due to incorrect paths or non-standard filenames. If you can't resolve it, paste the error message to the AI.
💡 More questions? See
FAQ.md— contains a complete usage guide, common questions, and troubleshooting.