Nexus Testing

Nexus Flaky Test Root Cause Analyzer

Structured protocol for diagnosing, isolating, and permanently eliminating non-deterministic test failures. Not a retry wrapper — treats flakiness as a first-class defect.

name:             nexus-testing
category:         testing / quality
required_context: test file path, failure frequency, CI log or local output, stack trace if available
expected_inputs:  test name, framework, failure pattern, environment (local/CI/both), reproduction rate
expected_outputs: flakiness type, reproduction steps, root cause (one sentence), narrowest fix +
                  verification command, prevention recommendation

Core Principles

1. A test failing 1-in-20 runs is still failing — it is a false alarm factory.
2. A retry without investigation ships unknown risk.
3. Fix the root cause, not the symptom — masking a race condition makes it fail more often.
4. Never mark xfail/skip without a linked issue and expiry date.
5. Test flakiness often mirrors a real production concurrency or data integrity hazard.

---

Step 1 — Collect Context

Required before any investigation:

| Signal | How | |--------|-----| | Failure rate | Estimate from CI history (1/5? 1/100?) | | Stack trace | Full verbatim trace from a failing run | | Test path | Exact: tests/users/test_create.py::test_create_user | | Framework + version | pytest --version, jest --version, go version | | CI vs local | Fails only in CI, only locally, or both? | | Parallelism config | -n auto, --workers, t.Parallel()? | | Recent changes | git log --oneline -10 on test file and its imports |

If CI-only failure, diff these before reading code: runtime version, OS/arch, parallelism, TZ / NTP, network access, env vars (missing vars silently produce defaults), I/O speed, Docker layer cache.

---

Step 2 — Classify Flakiness Type

| Type | Signature | Mechanism | |------|-----------|-----------| | Ordering | Fails after specific other tests; passes alone | Shared mutable state not reset | | Timing | Fails on slow machines or under load | Hard-coded delays, no backoff, wall-clock assertions | | Resource | Fails parallel, passes single-threaded | Port conflicts, shared temp dirs, DB row locks | | Environment | Fails in CI only | Runtime delta, missing env var, OS behavior | | External | Network errors, connection refused, timeouts | Real HTTP/DB calls in test | | Concurrency | Assertion error varies each run; stack trace differs | Thread/async race, missing lock |

Heuristic:

• Fails when run alone → Timing / Resource / External / Concurrency
• Passes when run alone → Ordering (shared state from another test)
• Fails only in CI → Environment delta
• Failure message varies → Concurrency or External
• Fails more under parallel load → Resource contention or Concurrency

Record: Primary: | Secondary:

---

Step 3 — Reproduce

Run at minimum 5 times before investigating. Confirm flaky, not consistently failing.

pytest tests/path/to/test.py::test_fn -xvs                          # alone, verbose
pytest tests/ --randomly-seed=last -x                                # ordering check
pytest tests/ -n auto -x                                             # contention check
for i in {1..20}; do pytest tests/path/to/test.py::test_fn -x --tb=no -q; done | grep -c FAILED

go test ./pkg/... -run TestFn -count=20 -v 2>&1 | grep -E "PASS|FAIL"  # Go
jest --testNamePattern "name" --runInBand --verbose                       # Jest

Record: X/N failed, fails alone or suite-only, rate under parallelism, verbatim stack trace.

---

Step 4 — Identify Root Cause

Ordering: find unreset shared state — grep -n "global\|@classmethod\|setUp\|tearDown\|autouse"; look for DB rows not rolled back, in-memory caches, module-level singletons, scope="session" fixtures.

Timing: grep -rn "time.sleep\|asyncio.sleep\|setTimeout" and polling loops without timeout; look for fixed delays insufficient on slow machines, wall-clock assertions.

Resource: grep -rn "port.*=[0-9]\{4,5\}\|localhost:[0-9]\{4,5\}" and grep -rn "tmp\|tempfile"; look for hardcoded ports, shared temp dirs, DB connections without rollback.

Environment: cat .github/workflows/*.yml | grep -E "python-version|node-version|go-version"; compare env vars between local and CI.

External: grep -rn "requests\.\|httpx\.\|fetch(\|http.Get" and real DB connection strings in tests.

Concurrency: grep -rn "threading\.\|asyncio\.\|goroutine" and missing Lock/Mutex/await.

State root cause as exactly one sentence: > "test_X fails intermittently because Y when Z."

If you cannot state it in one sentence, root cause is not yet identified.

---

Step 5 — Fix

Apply the narrowest fix. Do not improve unrelated code.

| Type | Wrong fix | Correct fix | |------|-----------|-------------| | Ordering | Delete the interfering test | Add teardown resetting shared state after each test | | Timing | Increase time.sleep(N) | Replace with wait_for(condition, timeout=N) | | Resource | Manually assign different port | Use port=0 or a free_port() fixture | | Environment | Hard-code CI env locally | Parametrize via env var; test both modes in CI | | External | Add retry in test | Mock external call; test real integration separately | | Concurrency | Add sleep before assertion | Use lock, event, barrier, or join |

Verification — required consecutive passes by failure rate:

| Rate | Required passes | |------|----------------| | >20% | 20 consecutive passes | | 5–20% | 30–50 consecutive passes | | 1–5% | 50 consecutive passes | | <1% | 100 passes or 30-day CI monitoring |

for i in {1..20}; do pytest tests/path/to/test.py::test_fn -x --tb=short -q; done

---

Step 6 — Prevention

• Ordering: add pytest --randomly-seed=random to every PR; fixture teardown in PR template.
• Timing: add wait_for(condition, timeout, poll_interval) helper; flag new time.sleep in CI.
• Resource: add free_port() and tmp_dir() fixtures to conftest.py.
• External: enforce --block-network in unit tests; separate @pytest.mark.integration.
• Concurrency: shared AsyncEventLoop fixture; asyncio_mode = "auto" in pytest.ini.

---

Output Contract

Every investigation closes with this report. Fill every field.

## Flaky Test Report

**Test:**             [full path + test name]
**Framework:**        [pytest / jest / go test / other]
**Failure Rate:**     [X/N runs]
**Primary Type:**     [Ordering | Timing | Resource | Environment | External | Concurrency]
**Secondary Type:**   [same options, or "none"]
**Root Cause:**       [one sentence — "test_X fails because Y when Z"]
**Why It Happens:**   [2–3 sentences — mechanism]
**Fix Applied:**      [file path + line numbers + change description]
**Verification:**     [command + result — "20/20 passes"]
**Prevention:**       [guardrail that stops this class from returning]
**Follow-up Needed:** [yes/no — if yes, describe]

---

Anti-Patterns (Quick Reference)

• Do not add time.sleep() — masks the race, breaks on slower machines.
• Do not mark xfail/skip without a linked issue and target fix date.
• Do not retry in CI without investigating — retries hide failures.
• Do not close without a reproduction command in the ticket.
• Do not declare fixed after < required consecutive passes.

---

Sub-documents

| Document | Purpose | |----------|---------| | checklists/investigation-checklist.md | Pre/during/post checklists for structured investigation | | anti-patterns/common-mistakes.md | Common wrong fixes and why they fail | | validation/output-validation.md | Fix confirmation, confidence scoring, escalation |

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: aayushostwal
• Source: aayushostwal/nexus
• License: MIT
• Homepage: https://nexus-ai.aayushostwal.com

Install and usage instructions live in the source repository linked above.