Test Reliability

A retried test that still flakes will eventually fail 3-of-3 during your most critical release, and a silently auto-repaired test may now verify a different element entirely. This skill builds suites teams can trust: resilient locators, classified flakes, environment-aware healing, data healing, and observable repair with confidence scoring — every automated fix produces evidence a human can review.

Quick Route

| Situation | Go to | |-----------|-------| | Test flakes in CI, root cause unknown | Flake Classification → decision tree | | Locator broke, want resilient replacement | Locator Resilience | | Action fails, suspect slow backend not UI bug | Environment-Aware Healing | | 401/404/429 mid-test from stale data | Data Healing | | Want auto-repair with review gate | Observable Repair Workflow | | Isolate a flaky test without blocking CI | Quarantine Management | | Step-by-step triage of one flaky test | references/flaky-test-runbook.md |

Discovery Questions

Check .agents/qa-project-context.md first — it carries known flaky areas, selector strategy, and CI environment details. Skip any question it already answers.

• What is your current flaky test rate? Check CI failure stats over the last 30 days. Below 2% is healthy; 2-5% needs attention; above 5% is eroding team trust.
• Where is the pain concentrated? Locator breakage? Timing? Test data? Environment? If unknown, instrument first (see Flake Classification).
• What is your current selector strategy? data-testid everywhere, mixed CSS and role-based, or no strategy (whatever works)? This sets the stability-score baseline.
• How do you handle flaky tests today? Retry and hope, skip and forget, or something structured? Decides how much process you need to add.
• What CI environment runs the tests? Same machine every time or different runners? Consistent or variable resources? Drives the environment-vs-test diagnosis.
• What is your test data strategy? Shared database, per-test fixtures, factory seeding, or external services? Decides whether data healing applies.

Core Principles

1. Prevention over cure. Writing a resilient test costs 1x. Investigating a flaky one costs 10x. Losing team trust in the suite costs 100x.

1. Healing must be observable and reviewable. Every automated repair produces evidence: what broke, what was tried, what worked, the confidence score. Silent fixes erode trust as fast as silent failures.

1. Classify before fixing. The fix for a timing issue is completely different from the fix for a data dependency. Wrong diagnosis wastes effort and can make things worse.

1. Flaky tests are bugs. Not annoyances to tolerate. A flaky test either has a test bug (fix the test), reveals an app bug (fix the app), or exposes an environment issue (fix the environment).

1. Track reliability as a metric, not a feeling. Measure flaky rate, mean time to heal, quarantine age, and selector stability. What gets measured gets fixed.

1. Self-healing is a spectrum. Start with resilient locators (Level 1), add fallback strategies (Level 2), then environment-aware healing (Level 3), then confidence-scored auto-repair (Level 4). Do not jump to Level 4 before mastering Level 1.

Locator Resilience

Multi-Attribute Selectors (Beyond Fallback Chains)

A single locator strategy is a single point of failure. Multi-attribute selectors combine multiple signals for one element lookup — resilience without fallback-chain complexity.

The key insight: instead of "try A, then B, then C," use "find element matching A AND B AND C with tolerance for one signal missing."

// Multi-attribute locator: tries combinations from most specific to least
const submitBtn = await multiAttributeLocator(page, {
  testId: 'checkout-submit',              // most stable signal
  role: 'button',                          // semantic signal
  name: /place order/i,                    // accessible name
  nearText: 'Order Summary',              // visual context
});
// Internally: tries testId+role+name first, then testId alone, then role+name,
// then text, then nearText+role. Returns first visible match.
// Unlike fallback chains, it combines signals for higher confidence.

DOM Similarity / Neighbor Context Matching

When a locator fails, the element may still exist with changed attributes. Use surrounding DOM context to find it:

1. Parent + tag + type: Find the container (by testId), then locate by tag and type within it.
2. Preceding label: Find sibling text (label), then locate the adjacent input/button.
3. Nearby text context: Find visible text near the target, then locate the element type in the same parent.

These are repair candidates scored by the confidence system below — not runtime fallbacks.

Selector Stability Scoring

Rate every selector on a 0-5 scale to prioritize refactoring.

| Score | Strategy | Survives | |-------|----------|----------| | 5 | getByTestId('submit-order') | CSS, text, and structural changes | | 4 | getByRole('button', { name: 'Submit' }) | CSS and structural changes | | 3 | getByLabel('Email') | CSS changes; breaks on label rewording | | 2 | getByText('Submit Order') | Breaks on any copy change | | 1 | locator('.btn-primary.submit') | Breaks on CSS or structural change | | 0 | locator('//div[3]/button[1]') | Breaks on any DOM change |

Target: Average score of 3.5+ across the suite. Audit monthly. Prioritize fixing score-0 and score-1 selectors. Emit one score per locator to selector-stability.md (or a CI step) and report the suite average so the 3.5 target is verifiable, not asserted.

Flake Classification Framework

Every flaky test has a root cause category. Classifying correctly determines the fix.

Categories

| Category | Signal | Root Cause | Fix Direction | |----------|--------|------------|---------------| | Timing | Timeout errors, passes on retry, worse in CI | Race condition, animation, async operation | Wait for condition, not time | | Data dependency | Fails with other tests, passes alone | Shared state, missing cleanup | Isolate per-test, fixture cleanup | | Environment | Fails on specific runner, correlates with load | Resource contention, network latency | Mock externals, increase resources | | Order dependency | Fails with --shard or fullyParallel | Depends on another test's side effect | Self-contained setup | | Time sensitivity | Fails at specific times (midnight, month-end) | Uses real clock, date boundary | Mock clock, relative comparisons | | Visual rendering | Screenshot diff flickers, subpixel differences | Font rendering, antialiasing, animation frame | Increase threshold, mask dynamic regions | | External service | Correlates with third-party status | Real HTTP calls in tests | Mock external APIs |

Classification Decision Tree

Test is flaky
│
├── Does it pass when run alone?
│   ├── YES → ORDER DEPENDENCY or DATA DEPENDENCY
│   │   ├── Does another test create/modify data it needs? → ORDER DEPENDENCY
│   │   └── Does it share a database/file/cache? → DATA DEPENDENCY
│   │
│   └── NO → Not order/data dependent. Continue below.
│
├── Does it fail more often in CI than locally?
│   ├── YES → TIMING or ENVIRONMENT
│   │   ├── Timeout errors? → TIMING (CI is slower)
│   │   ├── Connection errors? → ENVIRONMENT (network latency / service)
│   │   └── Resource errors (OOM, disk)? → ENVIRONMENT (resource contention)
│   │
│   └── NO → Same rate locally and CI. Continue below.
│
├── Does it fail at specific times?
│   ├── YES → TIME SENSITIVITY
│   │   ├── Near midnight? → Date boundary issue
│   │   ├── Near month/year end? → Calendar calculation
│   │   └── Specific hour? → Timezone issue
│   │
│   └── NO → Continue below.
│
├── Does it involve screenshots or visual comparison?
│   ├── YES → VISUAL RENDERING
│   │
│   └── NO → Continue below.
│
├── Does it call external HTTP APIs?
│   ├── YES → EXTERNAL SERVICE
│   │
│   └── NO → TIMING (most likely — default classification)
│       └── Investigate: what async operation is not being awaited?

The fix per category lives in references/flaky-test-runbook.md (Step 4) with full code patterns.

Environment-Aware Healing

Not all test failures are test problems. Some are environment problems. Environment-aware healing distinguishes the two and adapts.

Slow Backend vs True UI Failure

When an action fails, check backend health before blaming the test:

1. Action fails → Hit /api/health.
2. Backend unhealthy (5xx or timeout) → Retry with exponential backoff. Diagnose as backend_down. This is not a UI bug.
3. Backend healthy (2xx) → This is a real UI/test failure. Do not retry.

Return a structured diagnosis: { success: boolean; diagnosis: 'backend_down' | 'ui_failure' | 'backend_slow_recovered' }. This feeds flake classification — backend issues are environment issues, not test bugs.

Resource Contention Detection

Before declaring a test failure in CI, check for resource contention:

• Browser health: Load about:blank. If it takes > 2s (baseline 5s (baseline {

// 1. Try to find existing test user by deterministic email // 2. Verify auth token is still valid (GET /api/me) // 3. If token expired → refresh it (POST /refresh-token), mark as healed // 4. If user missing → create new one, mark as healed // 5. If healed → annotate testInfo for observability // 6. use(user) → run the test // 7. Cleanup: delete test user (guaranteed by fixture, even on failure) }

**Key patterns:**
- Use `testInfo.testId` in email/identifiers for per-test uniqueness.
- Annotate `testInfo.annotations` when healing occurs, for observability.
- Always clean up in the fixture's post-use block, not in `afterEach` — fixtures guarantee cleanup on failure.

## Observable Repair Workflow

**Core guardrail:** Healing must be observable and reviewable. Every repair follows this flow:

Failure Detected │ ▼ Candidate Repair Generated │ ▼ Confidence Score Computed (0.0 - 1.0) │ ▼ Evidence Diff Produced (what changed, what was tried) │ ▼ Approval Policy Applied │ ├── Score >= 0.9 → Auto-apply, log for batch review │ ├── Score 0.7-0.89 → Apply in quarantine, flag for individual review │ ├── Score 0.5-0.69 → Do NOT apply, open PR with evidence for review │ └── Score = 0.9** — Auto-apply, log for batch review.

• 0.7-0.89 — Apply in quarantine, flag for individual review.
• 0.5-0.69 — Do not apply; open a PR with evidence for review.
• ** { / ... / });

```typescript
// playwright.config.ts — separate projects
projects: [
  { name: 'stable', testMatch: /.*\.spec\.ts/, grep: /^(?!.*@quarantine)/ },  // exclude quarantine
  { name: 'quarantine', grep: /@quarantine/, retries: 3 },
],

In CI, run --project=stable as a blocking step and --project=quarantine with continue-on-error: true so the quarantine project never blocks the pipeline.

Quarantine Lifecycle

1. DETECT    — Test identified as flaky (CI reporter or manual triage)
2. TAG       — Add @quarantine annotation with ticket link and date
3. ISOLATE   — Quarantine project runs separately, does not block
4. DIAGNOSE  — Follow the flaky test runbook (references/flaky-test-runbook.md)
5. FIX       — Apply the fix pattern for the classified category
6. VERIFY    — Run 50x with --repeat-each, zero failures required
7. RELEASE   — Remove @quarantine tag, add annotation documenting the fix

Quarantine Hygiene Rules

• Maximum quarantine age: 14 days. After 14 days, fix it or delete it. Permanent quarantine is permanent rot.
• Every quarantine entry has a ticket link. No anonymous quarantines.
• Weekly review. Check the quarantine list every sprint. Aging quarantines get escalated.
• Track quarantine size. More than 5% of tests in quarantine signals a systemic problem requiring process change, not just test fixes.

Anti-Patterns

1. Silent Selector Replacement

Replacing a broken selector with no logging, review, or confidence scoring. The repaired test may now verify a different element entirely. Every repair must produce evidence.

2. "Just Retry It" as a Fix

Retries are a detection mechanism, not a fix. A test that needs retry 2-of-3 will eventually fail 3-of-3 during your most critical release.

3. Disabling Flaky Tests Permanently

test.skip('flaky, will fix later') — "later" never comes. Either quarantine with tracking or delete entirely. Skipped tests with no ticket are dead code.

4. Treating All Flakiness the Same

Timing issues and data dependencies need completely different fixes. Adding waitForTimeout(5000) to a data-dependency problem makes the test slower and still flaky.

5. waitForTimeout as a Stability Fix

// NEVER the right fix
await page.waitForTimeout(5000);

// Wait for the actual condition
await expect(page.getByRole('table')).toBeVisible();
await page.waitForResponse(resp => resp.url().includes('/api/data') && resp.status() === 200);

6. Healing Without Observability

Auto-repair that produces no logs, evidence, or confidence scores. You cannot improve what you cannot measure, and you cannot trust what you cannot review.

7. Over-Engineering Healing Before Writing Stable Tests

Building a complex self-healing framework before adopting basic resilient-locator patterns. Start with multi-attribute selectors and proper waits. Add healing infrastructure only when data shows where breakage occurs.

8. No Quarantine Expiry

Tests sit in quarantine for months. Quarantine is a temporary state, not a permanent home. Enforce a 14-day maximum.

Failure Modes

| Symptom | Likely cause | Fix or check | |---------|--------------|--------------| | Backend health check itself flaps → false backend_down diagnosis | Health endpoint is itself flaky/slow | Track the health endpoint's own p99 separately; don't gate diagnosis on a single probe | | Artifact storage balloons after enabling repair video | page.screencast recording on every run, not just repairs | Record only on the repair path, not the happy path | | Auto-repair accuracy drops below 80% | Confidence threshold too low, or intent-fidelity check skipped | Raise the auto-apply floor; never skip the intent check | | Quarantine project blocks the pipeline | Missing continue-on-error on the quarantine CI step | Add it; the quarantine project must never block merges |

Verification

• Reproduce the flake: npx playwright test --repeat-each=20 --workers=4 --trace=on — it must fail at least once before you trust any fix.
• After fixing, prove stability: npx playwright test --repeat-each=50 --workers=4 — require 50/50 passes, in CI conditions too.
• Confirm quarantine routing: npx playwright test --project=stable excludes @quarantine tests and --project=quarantine runs only them.
• Confirm selector audit emits numbers: the stability report lists a score per locator and a suite average.

Done When

• Every flaky test is identified and categorized by root cause (timing, data dependency, environment, etc.).
• Each flaky test is quarantined or fixed — no test silently retried without a documented plan and ticket reference.
• selector-stability.md (or the CI report) lists a stability score per locator and reports a suite average >= 3.5.
• The flaky-test-rate metric (% of tests passing on retry) is published to the CI dashboard and visible to the team.
• Every quarantine entry has a ticket reference and an expiry date <= 14 days out.

Related Skills

• selector-drift-recovery — go there to bulk-regenerate many selectors offline after a planned UI refactor; this skill heals one test at runtime.
• playwright-automation — full Playwright setup, Page Object Model, fixtures, and CI integration that the patterns here build on.
• ci-cd-integration — pipeline configuration, parallel execution, and the quarantine job wiring re

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: petrkindlmann
• Source: petrkindlmann/qa-skills
• License: MIT
• Homepage: https://qa-skills.com

Install and usage instructions live in the source repository linked above.