Evidence-Driven Debugging & Performance Profiling
Solve complex defects and performance bottlenecks systematically using the scientific method—never shotgun edits or unmeasured micro-optimizations.
When to Use
- Investigating crashes, intermittent bugs, race conditions, deadlocks, memory/handle leaks, or data corruption.
- Diagnosing p99 latency spikes, N+1 queries, web jank/CLS/bundle bloat, mobile ANRs/battery drain, game frame-time/GC spikes, or infra OOMKills/cold starts.
Non-Negotiable Rules
- Read playbooks first: Load
references/profiling-playbooks.mdfor the active stack before proposing fixes. - Never guess-patch: Do not edit code until you have a concrete reproduction or hard telemetry/profiler evidence proving the root cause.
- Measure before AND after: Every performance optimization must report exact baseline metrics vs. post-fix metrics under identical conditions.
- Fix the root cause, not the symptom: Adding
sleep(), swallowing an exception, or slapping a cache over a broken O(N^2) query is prohibited. - Lock every fix with a regression test: Every bug fix must add an automated test that fails before the fix and passes after.
Workflow
- Capture Symptoms & Establish Baseline:
- Collect exact error messages, stack traces, logs, trace IDs, repro steps, environment/version details, and quantitative baseline numbers (e.g., p99 ms, frame time ms, heap MB, query cost).
- Create Minimal Deterministic Reproduction:
- Narrow the failure into the smallest failing test, CLI command, query, or benchmark harness.
- For intermittent/race bugs, run under stress loops, race detectors, or deterministic seeded replay.
- Instrument & Profile (Evidence Collection):
- Select the right profiler for the bottleneck (CPU flamegraph, heap/allocation profile, lock contention,
EXPLAIN (ANALYZE, BUFFERS), browser Performance/Memory panel, mobile systrace/Instruments, GPU/frame profiler). - Identify the exact hot path or failing invariant—cite file, line, and measured cost.
- Select the right profiler for the bottleneck (CPU flamegraph, heap/allocation profile, lock contention,
- Form & Test Ranked Hypotheses (1–3 max):
- State the primary hypothesis, what evidence would confirm it, and what evidence would falsify it.
- Verify via targeted diagnostic probe or bisect (
git bisectfor regressions) before modifying production logic.
- Apply Minimal Surgical Fix + Regression Test:
- Write the failing regression test (or benchmark assertion) first.
- Implement the cleanest root-cause fix upholding SOLID/KISS/ATOMIC principles.
- Verify before/after metrics and run the full verification suite +
engg-audit(Mode A) on the diff.
- Emit Post-Incident Note (for P0/P1 defects):
- Document Root Cause, Trigger Condition, Fix, Before/After Evidence, and Systemic Prevention (lint rule, invariant, or alert).
Gotchas
- Premature micro-optimization: Optimizing a function accounting for 0.4% of frame/request time while ignoring a 60% I/O stall or GC spike wastes time and harms readability.
- Observer effect: Heavy synchronous debug logging inside a race condition or hot loop can mask the very bug you are hunting.
- Dirtying the repo with debug prints: Remove all temporary instrumentation before committing, or promote genuinely useful signals into structured observability.