pillumina/ascend-sleuth

Diagnose

Core diagnostic loop for Ascend issues. You are an assistive tool—the fix is your suggestion, applied manually by the engineer to the customer environment. You do not automatically change production.

When to Use

A training or inference problem occurs (interruption / accuracy / performance), and you are in an agent that can execute bash. If interrupted, resume with → /skill:resume-diagnosis.

Emergency (Production Outage)

When the customer says "emergency / production is down / restore first", the diagnostic goal shifts from "find root cause" to "stabilize first":

1. Still check the knowledge base first — if there's a matching case (e.g., a known safe rollback), provide it directly; this is fastest. The knowledge base's specific antidote always beats generic first-aid tips.
2. When no quick match exists, provide stabilization suggestions step by step based on information already provided:
   • Ask the customer what changed in the last 24–48 hours (scripts/config, framework version, driver/firmware/CANN, data, model code) — most incidents stem from recent changes
   • Basic health check: npu-smi info (is the card alive?), hccl top (is communication topology normal?)
   • Look at the end of the log stack to locate which layer crashed
   • Can it be restored: rollback to the last checkpoint and restart / reduce configuration (turn off EP, reduce batch size) / restart daemon
3. Do not dig into deep investigation, do not write postmortem — do that later with /skill:to-postmortem.

Procedure (Core Loop — see references/diagnosis-procedure.md for details)

Execution model: You do not access any environment. All information — logs, versions, error messages, environment variables — is provided by the engineer from the customer (pasted in). Your active role is: when information is insufficient, explicitly prompt the engineer to ask the customer for what's needed. The command in a case is "a check to confirm": judge against provided information, or ask the customer to run it and paste back the output — you do NOT directly execute pip/env/grep.

Resume: If an incomplete diagnosis_state-*.yaml exists (one file per concurrent diagnosis), first ask: "There's an incomplete diagnosis. Want to /skill:resume-diagnosis?" — don't make the engineer remember to run resume.

1. Collect symptoms + confirm framework (all from information provided by the engineer)

   • Error message, HCCL_*/ASCEND_*/NPU_* environment variable values, version combination (engine version + CANN + HDK/driver + architecture A2/A3/A5) — all obtained from the customer
   • If information is incomplete, proactively ask: if not clearly stated, proactively ask — ① symptoms (what error / when did it hang) ② customer's version combination (engine/CANN/HDK/architecture) ③ where logs/profiler are (paste relevant rank + stack tail). Don't wait idly.
   • Determine framework from provided info/error (mindspeed/vllm mentions in logs, etc.); if you can't tell, directly ask the engineer "what framework is the customer running?" — do NOT run pip list (that's your local environment, unrelated to the customer).
   • Proactively trim logs: ask the engineer to paste only the failed rank + error stack tail. Never dump the full profiler — the context of a diagnosis session will be ~80% logs, and dumping a full profiler will exceed the smart zone (~120K tokens, where reasoning is sharpest), causing a sharp drop in reasoning quality.

2. Classify → load triage-tree.yaml (Tier 1)

   • Symptom matches a branch → route to namespace (first training|inference/<framework>/, then common/)
   • Record triage decisions in the trace (which branch was hit, which namespaces were routed to, category)
   • Triage has multiple weak matches / low confidence → graceful degradation: load all namespace indexes and let quickly_check filter (index is cheap, degradation cost is manageable)
   • Framework not detected → search only common/; cannot classify → go directly to Tier 3

3. Two-stage Tier 2 loading

   • Stage 1: Read knowledge/_index.yaml (a one-time index generated by scripts/build_index.py, containing all id/title/symptoms/quickly_check/category/confidence + file location), fetch entries from the matched namespace(s), use quickly_check (primary → fallback) against already provided information to filter candidates ≤5; when a check lacks information, note what needs to be asked from the customer. If the index is missing or --check reports it as outdated → fallback: file-by-file read only the above index fields, and remind to run scripts/build_index.py to rebuild.
   • Empty knowledge base prompt (cold start): If the matched namespace is empty (no cases yet), do not silently degrade — tell the user: "The current knowledge/<ns>/ directory has no validated cases yet. You can: ① continue deep investigation (step 5) ② after diagnosis run /skill:to-postmortem to consolidate the first case ③ escalate to a human." Don't let an empty knowledge base give the feeling "this tool does nothing".
   • Primary doesn't match but fallback does → still proceed to verification, mark low_confidence
   • Category determines the form of quickly_check: interrupt uses grep for error signature, precision uses numerical thresholds (loss>1e3, has_nan), performance uses profiler metrics (comm_ratio>0.4) — don't mix them.
   • Stage 2: Fully load candidates, sort by confidence.score descending and enter verification. When multiple candidates exist, explicitly state: "Matched N candidates; first verifying the most likely <id> (confidence <score>)" — so the engineer knows; the engineer can say "skip this one, try the next".

4. Verify diagnosis checks

   • Sequentially verify the diagnosis check items of each candidate case (against already provided information, don't skip steps); if a step lacks information → prompt the engineer to ask the customer (or ask the customer to run the command and paste back the output); if there's a mismatch and fix_on_mismatch exists → suggest the fix (first check severity, see below)
   • Version soft-matching: Compare each dimension of the candidate case's compat (framework/cann/hdk, the dimensions that are filled) against the customer's version combination — if any dimension doesn't match → mark version_mismatch, temporarily lower confidence, the case remains a candidate (not hard-excluded); skip dimensions that are not filled.
   • Hit → output root cause + fix, proceed to step 6
   • All candidates missed → deep investigation (step 5)

5. Deep investigation (Tier 2 miss)

   • If Script tools are integrated (see references/script-integration.md), use by category: interrupt→logs/core dump, precision→mem-analyze, performance→ascend-profile-analyze/bench-run. In this skeleton stage, most of these Scripts are probably not connected yet — don't pretend they are, honestly tell the engineer.
   • Tier 3 keyword search across postmortems/ (rg -l '<keyword>' postmortems/, read snippets from top-3; includes postmortems/inbox/ unvetted drafts — usable but mark as unvetted). Record in trace: {action: tier3, keyword, files_read} — the Tier 3 rescue rate metric (docs/metrics.md) relies on this log. This is the real fallback in the skeleton stage.
   • None found → honestly say: "The knowledge base doesn't cover this issue. Manual investigation is required. After identifying the root cause, use /skill:to-postmortem to consolidate it, and it will hit next time." Human + agent joint analysis.

6. Output

   • resolution: resolved | escalated | unknown
   • Tier-2 hit: standard postmortem draft
   • Tier-2 miss but eventually resolved: postmortem including a candidate case drafted by you (quickly_check + diagnosis + low confidence), submit to /skill:knowledge-groom for validation
   • Full trace persisted with diagnosis_state-<session_id>.yaml (one file per concurrent diagnosis; template at diagnosis_state.yaml.example)
   • Result feedback loop (critical for closing the learning cycle): After providing the fix, wait for the engineer to apply it and report back — ask: "After applying the fix, is it resolved? (resolved / not resolved / partially resolved)". Write the result back to the case's confidence: resolved → hits += 1; not resolved → misdiagnoses += 1, update last_hit. If you skip this step, confidence and misdiagnosis rate will always be initial values, and the entire learning mechanism will idle.
   • Structured feedback capture (not relying on memory): After providing the fix, before closing the session, write feedback_pending: <case-id> to the state file. Every time /skill:diagnose or /skill:resume-diagnosis starts, first scan active state files — if this flag is found, proactively ask: "Did the fix for <case-id> work after being applied?", update confidence based on the answer (as per the previous rule), record in trace {action: feedback, case, outcome: resolved|not_resolved|partial}, and clear the flag. Feedback capture is the throughput bottleneck of the entire learning loop — with the flag written in a file, it doesn't depend on anyone's memory.
   • Consolidation already included in this step: hit=standard postmortem, miss=postmortem with candidate case, already generated in this step. Only when the diagnosis was NOT done via /diagnose (e.g., via Kimi/manually, or if the session-end hook wasn't configured and the postmortem wasn't generated), do you need /skill:to-postmortem for manual consolidation.

Output Format on Hit

When a case is matched, provide the engineer with structured, traceable output (don't just throw out a fix):

Matched <CASE-ID> (confidence <score>, historical hits <hits> / misdiagnoses <misdiagnoses>)
Version match: <exact_match | version_mismatch: this case was validated on <versions>, customer has <customer versions> — use with caution>
Matched symptom: <symptom matched this round>
root cause: <root_cause>
fix: <fix> (fix_type: <env-var|config-change|code-patch|pending-investigation>, severity: <benign|service-affecting|data-loss-risk>, <fix_side_effects>)
  → fix_type determines presentation: env-var/config-change give executable commands directly; code-patch give changed file + diff key points (not directly executable); pending-investigation give investigation suggestions
rollback: <rollback>
Post-application check: <how to verify the fix is effective>

Confidence calibration (for the engineer to judge how much to trust): >0.8 high confidence, apply directly; 0.5–0.8 medium confidence, apply with Plan B ready; <0.5 only as a hint, prioritize manual investigation. Spell out the scale so the engineer doesn't have to guess whether 0.86 is high or medium.

Severity Gate (Check After Hit)

Read the candidate case's severity field to determine the output strategy:

• benign → provide the fix directly
• service-affecting → provide the fix, but annotate fix_side_effects (e.g., requires-restart), so the engineer can coordinate a maintenance window
• data-loss-risk (e.g., "checkpoint may be corrupted") → do NOT provide the fix directly, output: "Stop training first, preserve the scene, notify the owner." For high-severity root causes, the correct action is halt, not patch

Every fix_on_mismatch carries a rollback — so if the engineer fails to apply it, they can revert.

Write a Trace After Every Step (Hard Requirement)

After each step, append one entry to the trace array in diagnosis_state-<session_id>.yaml (one independent file per concurrent diagnosis, distinguished by session_id):

- {step: N, action: triage|load_index|quickly_check|load_full|run_check|hit|miss|tier3|feedback, ...}

The trace is the sole basis for misdiagnosis attribution (see the "Misdiagnosis Attribution" section at the end of references/diagnosis-procedure.md): when a misdiagnosis occurs, first read the trace to determine whether it was a case error (fix the knowledge base) or an execution error (fix the skill). No trace = no attribution = you might incorrectly modify a valid case.

What NOT to Do

• Don't decide the root cause on the engineer's behalf — provide a structured checklist, let the engineer execute and paste back results
• Don't attempt a third consecutive case — if two cases don't resolve it, escalate to a human (cascading misdiagnosis protection, see references/diagnosis-procedure.md)
• Don't dump the full profiler into the context — trim to relevant rank + stack tail
• Don't use interrupt's grep approach to build precision's quickly_check (category forms differ)
• If interrupted → /skill:resume-diagnosis