Flaky Test Analyzer

Flaky Test Analyzer

Purpose: Identify and resolve unstable automation tests.

---

When to Use

Use this skill when:

• A test passes and fails intermittently
• Test results are inconsistent across runs
• CI/CD pipeline has unreliable test results

---

Responsibilities

Detect flaky tests caused by:

• Unstable locators (dynamic classes, positional xpath)
• Timing issues (race conditions, slow page loads)
• Incorrect waits (hard sleep instead of smart waits)
• Environment dependency (data not cleaned up, external service down)
• Test data conflicts (shared data between parallel tests)

---

Analysis Workflow

1. Detect — Identify the failing test and reproduce the failure
2. Inspect — Read error logs, stack traces, and screenshots
3. Classify — Categorize the root cause (locator / timing / data / environment)
4. Fix — Apply the appropriate fix strategy
5. Verify — Re-run test multiple times to confirm stability

---

Common Flaky Causes & Fixes

Unstable Locator

Problem:

//div[3]/button
.css-1n2xyz-btn

Fix: Replace with stable locator following priority in .agents/rules/locator_strategy.md:

• id, data-testid, name, css selector (stable), xpath (relative)

---

Timing Issues

Problem:

Thread.sleep(3000);       // Hard sleep — BAD
page.waitForTimeout(2000); // Fixed delay — BAD

Fix: Use smart waits as defined in .agents/rules/selenium_rules.md and .agents/rules/playwright_rules.md:

// Selenium
WebDriverWait wait = new WebDriverWait(driver, Duration.ofSeconds(10));
wait.until(ExpectedConditions.visibilityOfElementLocated(By.id("result")));

// Playwright
await expect(page.getByRole('button', { name: 'Submit' })).toBeVisible();

---

Test Data Conflicts

Problem: Tests share mutable data → parallel runs conflict.

Fix: Use unique, traceable random data:

_@test.com

---

Stability Checklist

After fixing a flaky test, verify:

• [ ] Locator is unique and stable across reloads
• [ ] No hard sleep or fixed delays
• [ ] Test data is unique and deterministic
• [ ] Test is independent (no dependency on other tests)
• [ ] Test passes 5+ consecutive runs

---

Rules References

The agent MUST follow these rules when analyzing flaky tests:

• .agents/rules/locator_strategy.md — Locator stability rules
• .agents/rules/automation_rules.md — General automation best practices
• .agents/rules/selenium_rules.md — Selenium wait strategy
• .agents/rules/playwright_rules.md — Playwright auto-waiting

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: anhtester
• Source: anhtester/codex-testing-kit
• License: MIT
• Homepage: https://anhtester.com

Install and usage instructions live in the source repository linked above.