windchillscalanthes-ship-it/flaky-test-detective

Flaky Test Detective

Catch the test that passes and fails on the same code — and rewrite it so it's deterministic, before it trains your team to ignore red builds.

A flaky test is worse than no test. It fails once, someone reruns CI, it passes, and everyone learns that red doesn't mean broken. Soon a real regression sails through behind a "just rerun it." The cause is almost never the code under test — it's a hidden dependency on something the test doesn't control: the clock, the random seed, the order tests run in, a shared row in the database, a real network call, the iteration order of a map. This skill finds that hidden input, explains why it makes the test non-deterministic, and rewrites the test to control it.

When to use this

• A test fails intermittently — red on one run, green on the next, no code change.
• A test is green locally but red in CI (or vice versa), or only fails under parallelism or a particular order.
• Someone says "just rerun it," "it's flaky," or "passes on retry."
• You are writing or reviewing a test and want to catch non-determinism before it's committed.
• You want to harden a suite — find the latent flakiness before it fires.

If a test fails every time, it's not flaky — it's a real failure or a genuine bug; fix the code, don't stabilize the test. And never "fix" flakiness by adding a retry or a sleep that hides it — that buries the signal. The goal is a test that's deterministic, not one that's usually green.

Workflow

1. Establish context. Identify the language / framework / runner (pytest, Jest/Vitest, RSpec/Minitest, Go testing, JUnit, PHPUnit, …) and where it flakes — locally, in CI, only in parallel, only in a certain order, only at a certain time of day. Where and when it flakes is the biggest clue to the cause.

2. Reproduce the non-determinism. A flaky test you can't fail on demand isn't fixed, just quiet. Force it: run it many times, run it in random order, run it in isolation vs. with the full suite, pin it to one worker, or shift the clock/timezone. reference/detection.md has the exact commands per framework.

3. Classify the root cause against the catalog below, and assign a severity:

   • 🔴 Flaky — a concrete non-deterministic input reaches an assertion; it will fail eventually. Must be made deterministic.
   • 🟡 Latent — non-deterministic in principle but currently masked (a seed that happens to be stable, an order that happens to hold, a fast-enough CI). Name the condition that would break it.
   • 🟢 Deterministic — the test controls its inputs. No change.

4. Fix the root cause — don't paper over it. For each 🔴/🟡, give:

   • why it's non-deterministic — the exact uncontrolled input and how it reaches the assertion,
   • the deterministic rewrite as runnable code (framework-idiomatic), and
   • the result (e.g. "order-dependent → isolated" or "flaky 1-in-20 → green 1000×").

5. Report using the output template at the end. Lead with the 🔴s.

For language- and cause-specific depth, load the reference file relevant to the test in front of you:

• reference/root-causes.md — the eight root-cause families in depth: how each makes a test flaky, how to detect it, and the canonical fix.
• reference/frameworks.md — per-framework mechanics: controlling the clock, seeding RNG, randomizing/pinning test order, isolating state, and mocking the network in pytest, Jest/Vitest, RSpec/Minitest, Go, JUnit, and PHPUnit.
• reference/detection.md — how to reproduce flakiness on demand: rerun loops, order randomization, isolation/bisection, parallelism, and reading CI signals.
• reference/patterns.md — the deterministic-test principles: hermeticity, control the clock/RNG/network/order, per-test isolation, and the detect→fix→quarantine lifecycle.

Read the reference for the framework in front of you rather than guessing — the mechanics differ (freezing time, seeding, and order control are done completely differently in pytest vs. Jest vs. Go).

Root-cause catalog (the high-frequency offenders)

"Uncontrolled input" = anything the test's result depends on that the test does not fix. Flakiness is always an uncontrolled input reaching an assertion.

#	Root cause	Why it's non-deterministic	Deterministic fix
1	Test-order dependence / shared mutable state	A test relies on state left by another (a global, a class attr, a module singleton, an un-rolled-back row). Passes in one order, fails in another.	Isolate: fresh fixtures per test, reset globals in teardown, wrap each test in a transaction rollback. Randomize order in CI to catch it.
2	Time & timezone	now()/Date.now()/today() in the test or code — fails at midnight, month/year boundaries, under DST, or in CI's timezone.	Freeze the clock (freezegun/time-machine, Jest fake timers, timecop, Go injable clock); pin TZ; never assert against the real wall clock.
3	Unseeded randomness	random(), uuid4(), faker, shuffled data — a rare value trips an assertion once in N runs.	Seed the RNG deterministically per test; inject IDs/fakes; assert on properties, not a specific random value.
4	Async race conditions / sleep-based waits	sleep(100) then assert — works until CI is slower; unawaited promises; callbacks asserted before they run; event ordering.	Wait on the condition, not a duration (waitFor/await the actual signal); await every promise; use fake timers to control scheduling.
5	Real network / external services	A live HTTP/API/DNS call — fails on latency, rate limits, downtime, or changed data.	Mock at the boundary (responses/respx, nock/msw, VCR/WireMock); never hit a real network in a unit test.
6	Unordered-collection assertions	Asserting a specific order from a set/dict/map, a DB query without ORDER BY, os.listdir, or concurrent results.	Sort before comparing, assert set-equality/contains, or add an explicit ORDER BY. Don't depend on incidental ordering.
7	Leaked global state / resource bleed	Env vars, singletons, module caches, the CWD, open connections/files/ports mutated and not restored — pollutes later tests.	Save/restore around each test (fixtures/beforeEach/t.Cleanup); close every resource; monkeypatch with auto-undo.
8	Too-tight timeouts / perf assumptions	A hardcoded timeout or assert elapsed < X that holds on a fast laptop but not a loaded CI runner.	Remove wall-clock perf assertions from correctness tests; if a timeout is needed, make it generous and condition-based, not a tight constant.
9	Floating-point exact equality	assert result == 0.3 after arithmetic — rounding makes it occasionally unequal.	Assert with a tolerance (pytest.approx, toBeCloseTo, epsilon compare).
10	Locale / environment assumptions	Number/date/string formatting, case-folding, path separators, or filesystem case-sensitivity that differ across machines/CI.	Pin locale/encoding in the test; use locale-independent formatting; don't assert OS-specific paths.
11	Shared fixtures / DB not isolated	Tests share a database, cache, or temp dir and step on each other under parallelism.	Per-test transactions/rollback, unique schemas/namespaces per worker, tmp_path per test; no shared writable fixture.
12	Hidden concurrency in the code under test	Background threads, timers, or async tasks that finish (or don't) unpredictably relative to the assertion.	Make the boundary awaitable/injectable; drain/flush the queue deterministically; inject a controllable executor/clock.

The 🟡 tier is where flakiness hides before it fires: a test that's order-dependent but happens to run last, a seed that's currently stable, a sleep that's currently long enough. Name the condition that flips it to 🔴 — that's the fix waiting to happen.

Guiding principles

• Flakiness is an uncontrolled input, not bad luck. Every flaky test depends on something it doesn't fix: the clock, the seed, the order, the network, a shared row. Find that input and control it — freeze it, seed it, isolate it, mock it.
• Reproduce before you fix. If you can't make it fail on demand (rerun loop, random order, isolation), you haven't found the cause — you've hidden it.
• A test must own its state. It should pass alone, in any order, in parallel, a thousand times. Setup creates what it needs; teardown restores what it touched.
• Control the four usual suspects. Clock, randomness, network, and order are the source of most flakiness — make each one deterministic by default.
• Wait on conditions, never on time. sleep(n) is a race you'll lose on a slow runner. Await the actual signal or poll the condition with a generous cap.
• Never fix flakiness with a retry. Retries and blanket sleeps convert a visible defect into a silent one and destroy the suite's credibility.
• Quarantine, don't delete. If a flaky test can't be fixed immediately, mark it (skip/quarantine) with a tracking issue so it stops blocking — but keep it, and keep the signal that it's broken.
• Right-size. A genuinely order-independent, hermetic test needs none of this — don't bolt determinism theater onto a test that already controls its inputs.

Output template

## Flaky Test Review

**Stack:** <language / framework / runner>  ·  **Flakes:** <CI only / in parallel / random order / at time X>  ·  **Tests reviewed:** <n>

### 🔴 <file::test name> — FLAKY
**Root cause:** <order dependence / time / randomness / async race / network / ordering / global state / timeout / …>
**Why non-deterministic:** <the exact uncontrolled input and how it reaches the assertion>
**Reproduce:** <the command that makes it fail on demand — random order, rerun loop, isolation, clock shift>
**Deterministic rewrite:**
<runnable, framework-idiomatic code>
**Result:** <e.g. "order-dependent → isolated; green in 1000 random-order runs">

### 🟡 <file::test> — LATENT
**Root cause / Why it's currently masked / The condition that would flip it to flaky / Recommendation**

### 🟢 <file::test> — DETERMINISTIC
<one line: it controls its inputs — no change>

---
**Summary:** <n flaky, n latent.> <the worst uncontrolled input in one line.> <bottom line: stable to merge, or fix test X first.>

Always give the deterministic rewrite and the command to reproduce — a review that says "this is flaky" without the controlled-input fix and a way to prove it just hands the hard part back. Naming the uncontrolled input is the expertise the skill supplies.