Regression Testing

Forbidden invocations

NEVER invoke /sdd-* skills from this workflow. SDD is an optional user-installed ceremony; this skill ships self-contained and does not chain SDD under any condition. If you need to refactor KATA, fixtures, cli/, scripts/, or api/schemas/ pipeline, exit this skill first and invoke /framework-development — which itself runs Plan → Code → Verify → Archive natively (no SDD required).

This boundary is mechanical, not advisory: scripts/lint-skills.ts rejects any /sdd- mention outside this section. See: .claude/skills/agentic-qa-core/references/skill-composition-strategy.md §4 (governs users who manually install SDD).

Regression Testing — Execute, Analyze, Decide

Orchestrates the full release-readiness pipeline: trigger a CI suite, monitor it to completion, classify failures, score against release criteria, and emit a GO / CAUTION / NO-GO verdict plus a stakeholder report.

Three phases, always in this order: Execute → Analyze → Report. Do not skip analysis and jump to a report. Do not guess classification without reading failure logs.

---

Inputs

• .github/workflows/*.yml — workflow files for regression / smoke / sanity suites; defines triggers, inputs, and artifact uploads.
• .context/master-test-plan.md — regression Epic key + expected pass-rate SLOs per suite.
• playwright.config.ts — reporter config, retry policy, project matrix; needed to interpret retry counts and shard splits.
• Previous run's Allure report (artifact URL or local download under ./analysis/previous/) — baseline for trend computation.
• kata-manifest.json — registry of tests and ATCs available; used to cross-reference failed test IDs.
• .agents/jira-required.yaml — Jira refs (project key, work types, transitions) for filing regression issues.
• agentic-qa-core/references/defect-management-doctrine.md — canonical authority for classifying (Bug/Defect/Improvement), the mandatory field matrix, QA-Assignee ownership, and the QA process epic when a confirmed regression is filed in Jira (Phase 3). Read BEFORE filing any defect.

---

Subagent Dispatch Strategy

> Orchestration & Session contracts: this skill follows ./orchestration-doctrine.md (mandatory subagent dispatch — main thread is command center) AND ./session-management.md (Phase 0 resume check, plan-first persistence at .session///, archive on completion). Phase 0 (resume check) and Phase 1 (plan write) are NOT optional. The orchestrator also applies the per-stage Definition-of-Done gates in ./stage-gates.md: verify a stage's DoD BEFORE recording its progress checkpoint and advancing.

This skill is per-run scope: ` = - (e.g. staging-2026-05-20). Session state lives at .session/regression-testing//{plan.md, progress.md} per agentic-qa-core/references/session-management.md §3 + §9. The single highest-value resume case: if the Monitor subagent dies while watching a long CI run but RUN_ID was captured in plan.md, Phase 0 re-attaches via gh run view ` instead of re-triggering CI (saves 20–60 min of wall-clock).

This skill is compliant with the doctrine in CLAUDE.md §"Orchestration Mode (Subagent Strategy)" and the session contract in .claude/skills/agentic-qa-core/references/session-management.md. Every dispatch follows the 6-component briefing format defined in .claude/skills/agentic-qa-core/references/briefing-template.md, and the pattern selected per stage matches the decision guide in .claude/skills/agentic-qa-core/references/dispatch-patterns.md. The two CI-bound stages (long-running watch, multi-artifact download) and the high-volume failure classification step are the hotspots — everything else stays inline because the dispatch overhead is not justified.

| Stage | Pattern | Subagent role | |------------------------------------------------------------|------------|----------------------------------------------------------------------------------------------------------------| | Trigger workflow (gh workflow run) | Single | inline — no dispatch needed (one shell call) | | Wait/monitor gh run watch | Background | one Monitor subagent runs the watch; main thread continues with prep work; subagent notifies on exit | | Download 3 artifacts (allure / evidence / playwright) | Parallel | 3 simultaneous subagents, one per artifact; cap = 3 (no rate-limit risk) | | Classify failures (chunks of ~10 tests each) | Parallel | N subagents based on failure volume; cap = 10 to avoid context dilution | | Compute metrics (pass-rate, trends) | Single | inline — needs aggregated state, low cost | | Generate executive report | Single | inline — final synthesis, decisions live here | | GO / CAUTION / NO-GO verdict | Single | inline — main thread owns release decisions |

• Error protocol: On any subagent failure: STOP, report full context to user, present retry / skip / abort options. Do NOT auto-fix. See .claude/skills/agentic-qa-core/references/orchestration-doctrine.md.

---

Readiness Preflight Gate (MANDATORY — runs before Phase 0)

> Full doctrine: agentic-qa-core/references/preflight-gate.md. Runs FIRST, before the resume check and any gh workflow run. Two laws: (1) args-as-answers — the suite (regression/smoke/sanity), env, and any grep/test_file are provided args; ask only the gaps. (2) probe, don't assume. Surface gaps + REDs as ONE AskUserQuestion checklist; self-fix with approval + explanation; STOP on any blocking RED. This generalizes the Phase 1 §Preflight (gh auth) to a full readiness check pulled to t=0. Generic baseline (env resolution, secret/restart handling, the two laws, output contract) is inherited from the reference §3.1 — not repeated here. Below is only this skill's specific capability delta (note: test-user creds, MCPs and browsers live inside the CI runner, not the orchestrator).

| Capability | Need | Why here | |---|---|---| | GitHub CLI authenticated | REQUIRED | Every stage drives CI via gh (gh auth status, gh workflow run, gh run watch, gh run download). Not authed → user runs gh auth login (suggest the ! prefix); do not proceed. | | Workflow files present | REQUIRED | .github/workflows/ must hold the regression/smoke/sanity workflow for the chosen suite, with the inputs this skill passes. | | GitHub Actions Secrets/Variables | REQUIRED | The runner authenticates with env-prefixed creds (secrets._USER_EMAIL / _PASSWORD) + XRAY_* / ATLASSIAN_* as Repository/Environment Secrets — the suite 401s mid-run without them. gh secret list (add --env for environment scope) shows them; missing → gh secret set from .env. /adapt-framework only emits a manual list today, so this is the most common silent gap. | | Allure 3 local | REQUIRED | bunx allure resolves (devDep, no global install); allurerc.mjs present for bun allure:agent markdown triage. | | Active env | REQUIRED | The suite runs against > (default {{DEFAULT_ENV}}). Confirm it is the intended target before a 20–60 min run. | | [TMS_TOOL] (result sync) | OPTIONAL | Only when .agents/project.yaml testing.tms_cli is set — Stage 3 pushes run status. jira-xray → /xray-cli + XRAY_*. | | [ISSUE_TRACKER_TOOL] (file regression issues) | OPTIONAL | Only on NO-GO / CAUTION-with-regressions, to file issues. Load /acli then. |

Test-user creds, OpenAPI/API_TOKEN, DBHub and Playwright browsers live inside the CI runner, not the orchestrator — this skill does not exercise them locally, so they are out of scope for this gate. After the gate clears (all REQUIRED GREEN), continue to Phase 0 below.

---

Phase 0 — Session resume check (MANDATORY, inline)

Before suite selection or any gh workflow run, run the resume contract from agentic-qa-core/references/session-management.md §4:

1. Compute prospective ` = - from invocation context (env defaults to {{DEFAULT_ENV}}`).
2. Check .session/regression-testing//progress.md.
3. If it does NOT exist → proceed to suite selection + Phase 1 preflight + plan.md write.
4. If it DOES exist:

• Read plan.md (captured suite, env, workflow_file, RUN_ID if Phase 1 already triggered).
• Read tail of progress.md.
• If RUN_ID is present AND progress.md last entry is Phase 1 — Trigger — status: completed but Monitor entry is missing/failed: surface the option to re-attach to the existing RUN_ID via gh run view --json status,conclusion instead of re-triggering. This is the high-value resume case.
• Otherwise surface the standard offer resume / restart / abort. On restart, archive to .session/.archive/-regression-testing--aborted/ first.

---

When to run each suite

| Suite | Workflow file | Duration | Use when | |-------|---------------|----------|----------| | regression | regression.yml | 20-60 min | Pre-release validation, nightly full run | | smoke | smoke.yml | 2-5 min | Post-deploy health check, @critical only | | sanity | sanity.yml | 1-10 min | Validate one feature / one file / one grep pattern |

If the user says "run regression" with no qualifier, default to regression on {{DEFAULT_ENV}}. If they say "smoke" or "critical only", use smoke. If they specify a file, grep, or single feature, use sanity.

---

Local reporting (Allure 3, no global install)

Allure 3 is a devDep — bunx allure resolves to the local node_modules/.bin/allure, no brew install allure / scoop install allure required. Configuration lives at allurerc.mjs (Awesome plugin enabled).

| Use case | Script | Underlying command | |---|---|---| | Run tests + auto-generate report (human review) | bun allure:run | bunx allure run -- bun test | | Run tests + emit markdown for AI review | bun allure:agent | bunx allure agent -- bun test | | Generate report from existing ./allure-results | bun allure:generate | bunx allure generate ./allure-results | | Serve last generated report locally | bun allure:open | bunx allure open | | Live-refresh report during iterative dev | bun allure:watch | bunx allure watch ./allure-results |

bun allure:agent is the AI-friendly entry point: it produces a markdown summary the orchestrator (or a Verifier subagent) can read directly without parsing HTML. Use it whenever you need a structured pass/fail breakdown after a local re-run while triaging a CI failure (Phase 2 step 1, before downloading the merged-allure-results artifact from CI).

CI artifacts (merged-allure-results-{env}) are still produced by the workflow and downloaded via gh run download as documented in Phase 2 — those use the same allurerc config inside the runner.

---

Phase 1 — Execute

Preflight (always)

gh auth status
gh repo view --json name,owner
gh workflow list

If gh is not authenticated, stop and ask the user to run gh auth login. Do not proceed.

Write .session/regression-testing//plan.md per agentic-qa-core/references/session-management.md §6 BEFORE the Trigger step below. Capture: Goal (suite + env + reason for run), Inputs (workflow file path, env vars, optional grep/test_file for sanity), Approach (subagent pattern per stage from the dispatch table above), Phase breakdown (Trigger → Monitor → Download → Classify → Compute → Report → Verdict), Risks, Verification checklist (all 3 artifacts download + verdict emitted), Cross-references (.context/reports/regression--.md will hold the final verdict). RUN_ID lands in plan.md §Inputs AFTER the Trigger step captures it — append, do not rewrite the body.

Trigger

# Full regression
gh workflow run regression.yml \
  -f environment=staging \
  -f video_record=false \
  -f generate_allure=true

# Smoke
gh workflow run smoke.yml -f environment=staging -f generate_allure=true

# Sanity (grep OR test_file, never both)
gh workflow run sanity.yml -f environment=staging -f test_type=e2e -f grep="@auth"
gh workflow run sanity.yml -f environment=staging -f test_file="tests/e2e/auth/login.test.ts"

Capture run ID

# Wait 3-5 seconds for the run to register, then:
gh run list --workflow=regression.yml --limit=1 --json databaseId,status,createdAt -q '.[0].databaseId'

Store as RUN_ID. Every subsequent step uses it.

Progress checkpoint after Trigger: append RUN_ID to .session/regression-testing//plan.md §Inputs (so resume can re-attach) AND append a phase entry ## Phase 1.Trigger — with status: completed, next: Phase 1.Monitor, notes: RUN_ID= to progress.md. This is the critical persistence point — Trigger landing without RUN_ID persisted means resume cannot re-attach.

Monitor to completion

Use the dispatch defined in §Subagent Dispatch Strategy: Background. Delegate gh run watch to a Monitor subagent so the main thread is freed to prepare the report scaffold and load the classification rubric. See references/ci-cd-integration.md §"Monitoring the workflow run (Background dispatch)" for the full briefing.

Reference command (executed inside the subagent, not inline on the main thread):

gh run watch  --exit-status
# Fallback polling (only if gh run watch is unavailable):
gh run view  --json status,conclusion
# status: queued | in_progress | completed
# conclusion (only when completed): success | failure | cancelled | timed_out

Do not start Phase 2 until the Monitor returns status: completed.

Output of Phase 1

A short execution summary with: workflow name, run ID, environment, duration, conclusion, per-job status, artifact list, and the Allure URL pattern https://{owner}.github.io/{repo}/{environment}/{suite}/.

Read references/ci-cd-integration.md when configuring new workflows, debugging CI-only failures, tuning sharding / retries / timeouts, or wiring up secrets and variables.

---

Phase 2 — Analyze

Step 1: Collect data

Use the dispatch defined in §Subagent Dispatch Strategy: Parallel for the three artifact downloads (allure / evidence / playwright). Fan out three subagents in a single tool-call block — each owns one artifact, writes to its own directory, and reports back when its download is verified. The metadata reads (gh run view) stay inline because they are short.

Reference commands (the metadata reads run inline; the three gh run download calls live inside the parallel subagents):

# Inline (main thread): full run context
gh run view  --json status,conclusion,jobs,createdAt,updatedAt,url,headBranch,event,actor

# Inline (main thread): failed logs only (much smaller than --log)
gh run view  --log-failed

# Inline (main thread): list artifacts so the parallel dispatchers know what to fetch
gh run view  --json artifacts --jq '.artifacts[].name'

# Parallel subagent A — allure results
gh run download  -n merged-allure-results-staging -D ./analysis/

# Parallel subagent B — failure evidence (screenshots, traces, videos)
gh run download  -n e2e-failure-evidence       -D ./analysis/evidence/

# Parallel subagent C — playwright HTML report
gh run download  -n e2e-playwright-report      -D ./analysis/playwright/

Each subagent uses the briefing shape in agentic-qa-core/references/briefing-template.md §"Parallel — Download 3 CI artifacts in regression-testing". Cap the fan-out at 3 — there are only ever three artifact strea

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: upex-galaxy
• Source: upex-galaxy/agentic-qa-boilerplate
• License: MIT

Install and usage instructions live in the source repository linked above.