# Test Reliability

> >-

- **Type:** Skill
- **Install:** `agentstack add skill-petrkindlmann-qa-skills-test-reliability`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [petrkindlmann](https://agentstack.voostack.com/s/petrkindlmann)
- **Installs:** 0
- **Category:** [Developer Tools](https://agentstack.voostack.com/c/developer-tools)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [petrkindlmann](https://github.com/petrkindlmann)
- **Source:** https://github.com/petrkindlmann/qa-skills/tree/main/skills/test-reliability
- **Website:** https://qa-skills.com

## Install

```sh
agentstack add skill-petrkindlmann-qa-skills-test-reliability
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

A retried test that still flakes will eventually fail 3-of-3 during your most critical release, and a silently auto-repaired test may now verify a different element entirely. This skill builds suites teams can trust: resilient locators, classified flakes, environment-aware healing, data healing, and observable repair with confidence scoring — every automated fix produces evidence a human can review.

## Quick Route

| Situation | Go to |
|-----------|-------|
| Test flakes in CI, root cause unknown | Flake Classification → decision tree |
| Locator broke, want resilient replacement | Locator Resilience |
| Action fails, suspect slow backend not UI bug | Environment-Aware Healing |
| 401/404/429 mid-test from stale data | Data Healing |
| Want auto-repair with review gate | Observable Repair Workflow |
| Isolate a flaky test without blocking CI | Quarantine Management |
| Step-by-step triage of one flaky test | `references/flaky-test-runbook.md` |

## Discovery Questions

Check `.agents/qa-project-context.md` first — it carries known flaky areas, selector strategy, and CI environment details. Skip any question it already answers.

- **What is your current flaky test rate?** Check CI failure stats over the last 30 days. Below 2% is healthy; 2-5% needs attention; above 5% is eroding team trust.
- **Where is the pain concentrated?** Locator breakage? Timing? Test data? Environment? If unknown, instrument first (see Flake Classification).
- **What is your current selector strategy?** data-testid everywhere, mixed CSS and role-based, or no strategy (whatever works)? This sets the stability-score baseline.
- **How do you handle flaky tests today?** Retry and hope, skip and forget, or something structured? Decides how much process you need to add.
- **What CI environment runs the tests?** Same machine every time or different runners? Consistent or variable resources? Drives the environment-vs-test diagnosis.
- **What is your test data strategy?** Shared database, per-test fixtures, factory seeding, or external services? Decides whether data healing applies.

## Core Principles

1. **Prevention over cure.** Writing a resilient test costs 1x. Investigating a flaky one costs 10x. Losing team trust in the suite costs 100x.

2. **Healing must be observable and reviewable.** Every automated repair produces evidence: what broke, what was tried, what worked, the confidence score. Silent fixes erode trust as fast as silent failures.

3. **Classify before fixing.** The fix for a timing issue is completely different from the fix for a data dependency. Wrong diagnosis wastes effort and can make things worse.

4. **Flaky tests are bugs.** Not annoyances to tolerate. A flaky test either has a test bug (fix the test), reveals an app bug (fix the app), or exposes an environment issue (fix the environment).

5. **Track reliability as a metric, not a feeling.** Measure flaky rate, mean time to heal, quarantine age, and selector stability. What gets measured gets fixed.

6. **Self-healing is a spectrum.** Start with resilient locators (Level 1), add fallback strategies (Level 2), then environment-aware healing (Level 3), then confidence-scored auto-repair (Level 4). Do not jump to Level 4 before mastering Level 1.

## Locator Resilience

### Multi-Attribute Selectors (Beyond Fallback Chains)

A single locator strategy is a single point of failure. Multi-attribute selectors combine multiple signals for one element lookup — resilience without fallback-chain complexity.

**The key insight:** instead of "try A, then B, then C," use "find element matching A AND B AND C with tolerance for one signal missing."

```typescript
// Multi-attribute locator: tries combinations from most specific to least
const submitBtn = await multiAttributeLocator(page, {
  testId: 'checkout-submit',              // most stable signal
  role: 'button',                          // semantic signal
  name: /place order/i,                    // accessible name
  nearText: 'Order Summary',              // visual context
});
// Internally: tries testId+role+name first, then testId alone, then role+name,
// then text, then nearText+role. Returns first visible match.
// Unlike fallback chains, it combines signals for higher confidence.
```

### DOM Similarity / Neighbor Context Matching

When a locator fails, the element may still exist with changed attributes. Use surrounding DOM context to find it:

1. **Parent + tag + type:** Find the container (by testId), then locate by tag and type within it.
2. **Preceding label:** Find sibling text (label), then locate the adjacent input/button.
3. **Nearby text context:** Find visible text near the target, then locate the element type in the same parent.

These are repair candidates scored by the confidence system below — not runtime fallbacks.

### Selector Stability Scoring

Rate every selector on a 0-5 scale to prioritize refactoring.

| Score | Strategy | Survives |
|-------|----------|----------|
| 5 | `getByTestId('submit-order')` | CSS, text, and structural changes |
| 4 | `getByRole('button', { name: 'Submit' })` | CSS and structural changes |
| 3 | `getByLabel('Email')` | CSS changes; breaks on label rewording |
| 2 | `getByText('Submit Order')` | Breaks on any copy change |
| 1 | `locator('.btn-primary.submit')` | Breaks on CSS or structural change |
| 0 | `locator('//div[3]/button[1]')` | Breaks on any DOM change |

**Target:** Average score of 3.5+ across the suite. Audit monthly. Prioritize fixing score-0 and score-1 selectors. Emit one score per locator to `selector-stability.md` (or a CI step) and report the suite average so the 3.5 target is verifiable, not asserted.

## Flake Classification Framework

Every flaky test has a root cause category. Classifying correctly determines the fix.

### Categories

| Category | Signal | Root Cause | Fix Direction |
|----------|--------|------------|---------------|
| **Timing** | Timeout errors, passes on retry, worse in CI | Race condition, animation, async operation | Wait for condition, not time |
| **Data dependency** | Fails with other tests, passes alone | Shared state, missing cleanup | Isolate per-test, fixture cleanup |
| **Environment** | Fails on specific runner, correlates with load | Resource contention, network latency | Mock externals, increase resources |
| **Order dependency** | Fails with --shard or fullyParallel | Depends on another test's side effect | Self-contained setup |
| **Time sensitivity** | Fails at specific times (midnight, month-end) | Uses real clock, date boundary | Mock clock, relative comparisons |
| **Visual rendering** | Screenshot diff flickers, subpixel differences | Font rendering, antialiasing, animation frame | Increase threshold, mask dynamic regions |
| **External service** | Correlates with third-party status | Real HTTP calls in tests | Mock external APIs |

### Classification Decision Tree

```
Test is flaky
│
├── Does it pass when run alone?
│   ├── YES → ORDER DEPENDENCY or DATA DEPENDENCY
│   │   ├── Does another test create/modify data it needs? → ORDER DEPENDENCY
│   │   └── Does it share a database/file/cache? → DATA DEPENDENCY
│   │
│   └── NO → Not order/data dependent. Continue below.
│
├── Does it fail more often in CI than locally?
│   ├── YES → TIMING or ENVIRONMENT
│   │   ├── Timeout errors? → TIMING (CI is slower)
│   │   ├── Connection errors? → ENVIRONMENT (network latency / service)
│   │   └── Resource errors (OOM, disk)? → ENVIRONMENT (resource contention)
│   │
│   └── NO → Same rate locally and CI. Continue below.
│
├── Does it fail at specific times?
│   ├── YES → TIME SENSITIVITY
│   │   ├── Near midnight? → Date boundary issue
│   │   ├── Near month/year end? → Calendar calculation
│   │   └── Specific hour? → Timezone issue
│   │
│   └── NO → Continue below.
│
├── Does it involve screenshots or visual comparison?
│   ├── YES → VISUAL RENDERING
│   │
│   └── NO → Continue below.
│
├── Does it call external HTTP APIs?
│   ├── YES → EXTERNAL SERVICE
│   │
│   └── NO → TIMING (most likely — default classification)
│       └── Investigate: what async operation is not being awaited?
```

The fix per category lives in `references/flaky-test-runbook.md` (Step 4) with full code patterns.

## Environment-Aware Healing

Not all test failures are test problems. Some are environment problems. Environment-aware healing distinguishes the two and adapts.

### Slow Backend vs True UI Failure

When an action fails, check backend health before blaming the test:

1. **Action fails** → Hit `/api/health`.
2. **Backend unhealthy (5xx or timeout)** → Retry with exponential backoff. Diagnose as `backend_down`. This is not a UI bug.
3. **Backend healthy (2xx)** → This is a real UI/test failure. Do not retry.

Return a structured diagnosis: `{ success: boolean; diagnosis: 'backend_down' | 'ui_failure' | 'backend_slow_recovered' }`. This feeds flake classification — backend issues are environment issues, not test bugs.

### Resource Contention Detection

Before declaring a test failure in CI, check for resource contention:

- **Browser health:** Load `about:blank`. If it takes > 2s (baseline  5s (baseline  {
  // 1. Try to find existing test user by deterministic email
  // 2. Verify auth token is still valid (GET /api/me)
  // 3. If token expired → refresh it (POST /refresh-token), mark as healed
  // 4. If user missing → create new one, mark as healed
  // 5. If healed → annotate testInfo for observability
  // 6. use(user) → run the test
  // 7. Cleanup: delete test user (guaranteed by fixture, even on failure)
}
```

**Key patterns:**
- Use `testInfo.testId` in email/identifiers for per-test uniqueness.
- Annotate `testInfo.annotations` when healing occurs, for observability.
- Always clean up in the fixture's post-use block, not in `afterEach` — fixtures guarantee cleanup on failure.

## Observable Repair Workflow

**Core guardrail:** Healing must be observable and reviewable. Every repair follows this flow:

```
Failure Detected
  │
  ▼
Candidate Repair Generated
  │
  ▼
Confidence Score Computed (0.0 - 1.0)
  │
  ▼
Evidence Diff Produced (what changed, what was tried)
  │
  ▼
Approval Policy Applied
  │ ├── Score >= 0.9   → Auto-apply, log for batch review
  │ ├── Score 0.7-0.89 → Apply in quarantine, flag for individual review
  │ ├── Score 0.5-0.69 → Do NOT apply, open PR with evidence for review
  │ └── Score = 0.9** — Auto-apply, log for batch review.
- **0.7-0.89** — Apply in quarantine, flag for individual review.
- **0.5-0.69** — Do not apply; open a PR with evidence for review.
- ** { /* ... */ });
```

```typescript
// playwright.config.ts — separate projects
projects: [
  { name: 'stable', testMatch: /.*\.spec\.ts/, grep: /^(?!.*@quarantine)/ },  // exclude quarantine
  { name: 'quarantine', grep: /@quarantine/, retries: 3 },
],
```

In CI, run `--project=stable` as a blocking step and `--project=quarantine` with `continue-on-error: true` so the quarantine project never blocks the pipeline.

### Quarantine Lifecycle

```
1. DETECT    — Test identified as flaky (CI reporter or manual triage)
2. TAG       — Add @quarantine annotation with ticket link and date
3. ISOLATE   — Quarantine project runs separately, does not block
4. DIAGNOSE  — Follow the flaky test runbook (references/flaky-test-runbook.md)
5. FIX       — Apply the fix pattern for the classified category
6. VERIFY    — Run 50x with --repeat-each, zero failures required
7. RELEASE   — Remove @quarantine tag, add annotation documenting the fix
```

### Quarantine Hygiene Rules

- **Maximum quarantine age: 14 days.** After 14 days, fix it or delete it. Permanent quarantine is permanent rot.
- **Every quarantine entry has a ticket link.** No anonymous quarantines.
- **Weekly review.** Check the quarantine list every sprint. Aging quarantines get escalated.
- **Track quarantine size.** More than 5% of tests in quarantine signals a systemic problem requiring process change, not just test fixes.

## Anti-Patterns

### 1. Silent Selector Replacement
Replacing a broken selector with no logging, review, or confidence scoring. The repaired test may now verify a different element entirely. **Every repair must produce evidence.**

### 2. "Just Retry It" as a Fix
Retries are a detection mechanism, not a fix. A test that needs retry 2-of-3 will eventually fail 3-of-3 during your most critical release.

### 3. Disabling Flaky Tests Permanently
`test.skip('flaky, will fix later')` — "later" never comes. Either quarantine with tracking or delete entirely. Skipped tests with no ticket are dead code.

### 4. Treating All Flakiness the Same
Timing issues and data dependencies need completely different fixes. Adding `waitForTimeout(5000)` to a data-dependency problem makes the test slower and still flaky.

### 5. waitForTimeout as a Stability Fix

```typescript
// NEVER the right fix
await page.waitForTimeout(5000);

// Wait for the actual condition
await expect(page.getByRole('table')).toBeVisible();
await page.waitForResponse(resp => resp.url().includes('/api/data') && resp.status() === 200);
```

### 6. Healing Without Observability
Auto-repair that produces no logs, evidence, or confidence scores. You cannot improve what you cannot measure, and you cannot trust what you cannot review.

### 7. Over-Engineering Healing Before Writing Stable Tests
Building a complex self-healing framework before adopting basic resilient-locator patterns. Start with multi-attribute selectors and proper waits. Add healing infrastructure only when data shows where breakage occurs.

### 8. No Quarantine Expiry
Tests sit in quarantine for months. Quarantine is a temporary state, not a permanent home. Enforce a 14-day maximum.

## Failure Modes

| Symptom | Likely cause | Fix or check |
|---------|--------------|--------------|
| Backend health check itself flaps → false `backend_down` diagnosis | Health endpoint is itself flaky/slow | Track the health endpoint's own p99 separately; don't gate diagnosis on a single probe |
| Artifact storage balloons after enabling repair video | `page.screencast` recording on every run, not just repairs | Record only on the repair path, not the happy path |
| Auto-repair accuracy drops below 80% | Confidence threshold too low, or intent-fidelity check skipped | Raise the auto-apply floor; never skip the intent check |
| Quarantine project blocks the pipeline | Missing `continue-on-error` on the quarantine CI step | Add it; the quarantine project must never block merges |

## Verification

- Reproduce the flake: `npx playwright test  --repeat-each=20 --workers=4 --trace=on` — it must fail at least once before you trust any fix.
- After fixing, prove stability: `npx playwright test  --repeat-each=50 --workers=4` — require 50/50 passes, in CI conditions too.
- Confirm quarantine routing: `npx playwright test --project=stable` excludes `@quarantine` tests and `--project=quarantine` runs only them.
- Confirm selector audit emits numbers: the stability report lists a score per locator and a suite average.

## Done When

- Every flaky test is identified and categorized by root cause (timing, data dependency, environment, etc.).
- Each flaky test is quarantined or fixed — no test silently retried without a documented plan and ticket reference.
- `selector-stability.md` (or the CI report) lists a stability score per locator and reports a suite average >= 3.5.
- The flaky-test-rate metric (% of tests passing on retry) is published to the CI dashboard and visible to the team.
- Every quarantine entry has a ticket reference and an expiry date <= 14 days out.

## Related Skills

- **selector-drift-recovery** — go there to bulk-regenerate many selectors offline after a planned UI refactor; this skill heals one test at runtime.
- **playwright-automation** — full Playwright setup, Page Object Model, fixtures, and CI integration that the patterns here build on.
- **ci-cd-integration** — pipeline configuration, parallel execution, and the quarantine job wiring re

…

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [petrkindlmann](https://github.com/petrkindlmann)
- **Source:** [petrkindlmann/qa-skills](https://github.com/petrkindlmann/qa-skills)
- **License:** MIT
- **Homepage:** https://qa-skills.com

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** yes
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-petrkindlmann-qa-skills-test-reliability
- Seller: https://agentstack.voostack.com/s/petrkindlmann
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.