Test Automation Workflow

Test Automation Workflow — the IC process

This skill describes how individual contributors (analyst, implementer, reviewer) do their craft inside the analyst → implementer → reviewer pipeline. Orchestration of that pipeline is the test-automation-lead agent's job. That role owns slot routing, dispatch templates, AFS quality gating, status discipline, automation merge gate, and framework architecture decisions. This skill describes what each IC slot does once dispatched.

If you arrived here looking for routing / slot defaults / "when to involve tech-lead" / canonical dispatch prompts, read [agents/test-automation-lead/AGENT.md](agents/test-automation-lead/AGENT.md). On projects without test-automation-lead installed, those responsibilities fall to whichever agent has been substituted via .agents/role-overrides.md.

Core philosophy: do not automate what you have not executed. Every case is run manually first (pick whichever browser tool is wired and fits the challenge — playwright-testing over MCP, playwright-cli from the shell, browser-verify over CDP; full triage in [references/browser-tools.md](references/browser-tools.md)) so defects, missing data, and environmental gaps surface before a line of automation code is written. Then a separate engineer implements the automation inside the project's existing framework. Then a reviewer re-runs it.

Why split the work across slots: context. The analysis pass carries exploration state (DOM snapshots, test data, console noise). The automation pass carries framework state (page objects, fixtures, CI config). The review pass carries adversarial-eye state (assertion strength, masking suspicion). Cramming all of that into one session breaks the bot; the slot split keeps each workspace lean.

Orchestrator slot contract

This skill section IS the orchestrator-slot contract for the test-automation pipeline. When any agent is filling the orchestrator role — test-automation-lead by default, or any other agent named in .agents/team-comms.md § Roster — role, behavior, dispatch mechanics, and decision rules are fixed here so the role is portable: load this skill + point the roster at any orchestrator-capable agent.

Role. Route test-automation work through the analyst → implementer → reviewer pipeline, gate AFS quality, classify blockers and route them, own the automation merge, own test-framework architecture decisions.

Session context — read once at session start. Typically auto-imported via your agent's AGENT.md; if your agent doesn't auto-import, read them now:

• .agents/profile.md — project systems map (issue tracker, TMS, base branch, merge policy)
• .agents/workflow.md — branch/PR conventions, EPIC pattern, sub-task filing rules
• .agents/testing.md — framework, run commands, fixture/POM conventions, locator strategy, merge-gate N
• .agents/team-comms.md — host, dispatch syntax, installed roster
• .agents/role-overrides.md (if present) — slot substitutions when the default agent isn't installed

Missing files are tolerated except when ALL are absent — in that case the project isn't seeded; pause and ask the operator to run scout.

Per-batch parameters (caller / user provides):

• One or more TMS case IDs (or "all of SPRINT-42's regression suite", etc.)
• Implicit: .agents/profile.md § Automation PR policy (base branch, merge policy, merge strategy)

Return contract:

• After each meaningful turn: a status update (see playbook § Status reporting cadence)
• After each merge: tracker close + TMS back-write + user notification
• Escalations classified and routed (see playbook § Handling blockers)

Full playbook — dispatch mechanics, pre-flight checklists, canonical dispatch templates, AFS quality gate, status discipline, tracker discipline, status reporting, handling blockers, R2 cap rule, framework architecture (greenfield / framework-scale / mid-flow), merge protocol, anti-patterns — lives in [references/orchestration-playbook.md](references/orchestration-playbook.md). Load it once at session start.

Wiring this role on a project. To swap the default orchestrator for another agent (e.g. PM):

1. Add test-automation-workflow to the substitute agent's skills: frontmatter.
2. Update .agents/team-comms.md § Roster so the orchestrator slot points at the substitute.
3. The substitute now has the orchestrator slot contract + playbook access — the test-automation lead role is filled without test-automation-lead agent being installed.

Implementer slot contract

This skill IS the implementer slot in the test-automation pipeline. When dispatched — by an orchestrator like test-automation-lead, or standalone for "implement the AFS at ``" — role, context, parameters, and return shape are fixed here so dispatch prompts don't have to inline them.

Role. Take a ready-for-automation or extend-existing AFS, write the spec (and any required page-object / fixture changes), run it green N times locally, hand back a PR-ready diff plus a Run Report. Full mechanics in § Implementer six-phase loop and § Hard Rules — implementer below.

Session context — read once at session start. Typically auto-imported via @-blocks in your agent's AGENT.md; if your agent doesn't auto-import, read them now:

• .agents/profile.md — project systems, base URL, sample users
• .agents/workflow.md — branch/PR rules, commit authority
• .agents/testing.md — framework, run commands, locator strategy, POM conventions
• .agents/architecture.md — surfaces under test
• .agents/memory//project_briefing.md — accumulated project gotchas
• This skill's § Hard Rules — implementer (the forbidden list and additive-only rule)

Missing context → flag the gap; don't fabricate defaults.

AFS gate (refuse and return if violated — analyst output drives this):

• Accept: ready-for-automation (fresh spec) or extend-existing (extend the named covering spec per the AFS's Gap assertions section).
• Refuse: already-covered (no implementation needed — traceability AFS only), blocked, defect-found, un-automatable, out-of-scope-by-author.

Per-case parameters (caller provides at dispatch time):

• TMS case ID
• AFS path
• User set — a key into .agents/profile.md § Roles & sample users (e.g. ${TEST_USER})
• Branch name — if the caller created the branch. Don't switch, commit, push, or otherwise touch git unless .agents/workflow.md grants commit authority to this slot.

Retry budget. Soft limit: ≤ 2 reruns against the same root cause before escalating. The orchestrator's R2 cap rule will refuse R3 on the same cause regardless — see agents/test-automation-lead/AGENT.md § R2 cap rule.

Return contract:

• PR-ready diff (spec + page objects + fixtures in one commit set)
• Run Report per § Run Report — mandatory template (classification + evidence)
• If escalating after R2: name the class (architectural / AFS-drift / product-change) so the orchestrator routes correctly

The eight steps (IC view)

1. Discover framework          (read what scout produced)
2. Ingest case from TMS         (pluggable adapter)
3. Execute manually             (analyst slot via `test-case-analysis`)
4. Produce automation-ready spec (analyst emits AFS markdown)
5. Implement automation         (implementer six-phase loop — see § Implementer below)
6. Run & stabilize              (implementer — green or real defect; Run Report)
7. Review                       (reviewer slot + code-review skill)
8. Deliver & sync TMS           (completing-a-task + TMS adapter back-write)

Steps 1–4 belong to the analyst slot (driven by [test-case-analysis](skills/test-case-analysis/SKILL.md)). Steps 5–6 belong to the implementer slot (driven by this skill — see § Implementer six-phase loop below). Step 7 is the reviewer slot (driven by [code-review](skills/code-review/SKILL.md)). Step 8 is the handoff.

test-automation-lead resolves each slot to a concrete agent at dispatch time. ICs don't need to know the routing rules — they need to know how to execute their phase once dispatched.

1. Discover framework

Before anything, read what scout / seeding-a-project produced:

• AGENTS.md — tech stack, test commands
• .agents/testing.md — test framework, commands, fixtures, CI
• .agents/architecture.md — system map (for data flow awareness)
• .agents/profile.md — languages, default branch
• .agents/test-automation.yaml — TMS config + framework hints (if present)

If none of these exist, ask test-automation-lead to run scout first. Do not try to automate into a codebase you have not mapped.

Framework detection patterns (if testing.md doesn't name it):

test -f playwright.config.ts -o -f playwright.config.js && echo "playwright"
test -f cypress.config.ts -o -f cypress.config.js && echo "cypress"
find . -name "pom.xml" -maxdepth 3 -exec grep -l "selenium\|playwright" {} \;
grep -r "pytest-playwright\|playwright.sync_api" --include="*.txt" --include="*.toml" . 2>/dev/null | head
test -f wdio.conf.ts -o -f wdio.conf.js && echo "wdio"

No framework yet? Return needs-escalation (mid-flow escalation). The orchestrator owns the bootstrap decision per [agents/test-automation-lead/AGENT.md](agents/test-automation-lead/AGENT.md) § Framework Architecture. Once an approved plan is handed back, execute it against [references/framework-scaffold.md](references/framework-scaffold.md).

2. Ingest case from TMS

TMS is pluggable along two axes — adapter (which TMS) and transport (HTTP or MCP). The active combination is declared in .agents/test-automation.yaml (see [references/tms-adapters.md](references/tms-adapters.md)).

Supported adapters out of the box: zephyr-scale, testrail, xray, azure-test-plans, markdown (plain files). Each adapter exposes the same verbs regardless of transport:

fetch_case(id)       → returns { id, name, preconditions, steps, expected, cleanup, links }
update_execution(id, status, evidence) → back-writes result

Full-field fetch is mandatory. TMS adapters typically expose a "quick search" verb (returns minimal fields) and a "full fetch" verb (returns all custom fields including step + expected text). Always use full fetch. If your adapter has only quick-search and the step text comes back null, the case is unusable — stop and ask test-automation-lead how to get the full content (open the case in the browser and copy, if necessary). Never proceed on a partial case.

Transport choice:

• transport: mcp — preferred when the host has a TMS MCP server configured (Elitea, Atlassian Remote MCP, vendor TestRail / Xray MCP). The adapter calls mcp____ instead of issuing HTTP. Secrets live in the host's MCP config.
• transport: http — the TMS's public API with credentials from env vars. Works everywhere without host integration.

If no adapter is configured, default to markdown: cases live in test-specs/{feature}/l{priority}_{name}.md.

Never hardcode a TMS. All TMS logic flows through the adapter.

3. Execute manually (analyst slot)

The analyst — typically qa-engineer, occasionally a substitute — runs the case step-by-step against the real application, using the [test-case-analysis](skills/test-case-analysis/) skill:

• UI cases → [playwright-testing](skills/playwright-testing/) MCP tools, preferring browser_snapshot for accessible-name discovery.
• Fallback / deep inspection → [browser-verify](skills/browser-verify/) (CDP — real input events, computed styles, storage).
• API cases → curl / project's HTTP client.

For every step: screenshot, console, network. For every assertion: proof.

Output of this phase is truth, not code. What actually happened, not what the case says should happen.

4. Produce automation-ready spec (AFS)

The analyst writes an Automation-Friendly Spec (AFS) — a markdown file in test-specs/{feature}/l{priority}_{slug}_{tms-id}.md. Format and required sections live in [skills/test-case-analysis/references/spec-format.md](skills/test-case-analysis/references/spec-format.md).

AFS quality bar — implementer-readable contract. Every AFS must satisfy:

• User selection section — explicitly names the env var keys (e.g. ${TEST_USER} / ${TRIAL_USER} for projects with multi-credential sets).
• Test data inventory — three buckets: reuse-existing / generate-per-test / generate-shared-with-cleanup. Every datum classified.
• Stable selectors discovered, not guessed — every selector came from a real browser_snapshot or DOM inspection. Unobserved selectors marked "to-verify in implementer Phase 2 (Explore)".
• Known Defects Found — every defect filed with ticket ID + recommended handling (expect.soft() or natural-fail).
• Cleanup steps — state mutations + reset between runs.

An AFS missing any of these is blocked, not ready-for-automation. test-automation-lead enforces this gate before forwarding to the implementer.

If the case cannot be automated at all (e.g. physical card reader), the analyst says so explicitly and stops. Don't write automation for un-automatable cases.

Implementer six-phase loop

Absorb → Explore → Automate → Execute → Debug → Handoff. Six phases. Each ends with a checkpoint. Skip nothing.

Phase 1 — Absorb

Read the AFS end-to-end. Re-read .agents/testing.md. Open three neighbouring tests in the same feature area. Check the AFS Status field against the slot contract (§ Implementer slot contract above):

| Status | Action | |---|---| | ready-for-automation | Accept. Standard six-phase loop — write a fresh spec. | | extend-existing | Accept. Read the covering spec named in AFS § Extension target end-to-end AND read its own AFS (typically in the same test-specs// directory) so you know what's already proven. Then proceed through Phase 2–6 against AFS § Gap assertions only. The artefact you ship is an edit to the covering spec, not a fresh .spec.ts. | | already-covered | Refuse. No-implementation status — the lcovered_.md AFS is a traceability artefact only. Return to the orchestrator noting the misrouting. | | out-of-scope-by-author | Refuse. Analyst rejected at Phase 0 case-gate; should not reach implementer. Return to the orchestrator. | | blocked | Refuse. Report to the orchestrator with the unblock requirement. | | defect-found | Conditional. Confirm the defect ticket exists AND the AFS specifies handling (expect.soft() for isolated, let-it-fail-naturally for blocking). If unclear, refuse to the orchestrator. | | un-automatable | Refuse. Analyst should not have routed this. |

This table is the single source of truth. Orchestrator briefs, dispatch prompts, and project workflows defer to it. New statuses get added here first.

Phase 2 — Explore (skip if AFS selectors are confirmed against current DOM)

If your Absorb pass surfaces a discrepancy between AFS selectors and the live DOM (UI changed since analyst pass, or AFS noted "to-verify" selectors), explore before writing code:

1. Use the project's browser-driving capability (playwright-cli codegen, playwright-testing MCP, or browser-verify for computed styles).
2. Diff observed selectors vs AFS-stated selectors.
3. Amend the AFS in-place with a docs(afs): amend selectors per implementer exploration commit — do NOT silently drift from the AFS.
4. If the gap is too wide (multiple steps obsolete, app flow changed), return needs-analyst-rerun to the orchestrator — re-exploration is the analyst's job, not yours.

Phase 2 has a budget: 30 minutes of exploration before escalating to the orchestrator.

For extend-existing AFS: Phase 2 has an additional pre-step — read the covering spec end-to-end AND its own AFS (the one that authored it) before driving the live surface. The goal is to enter Phase 3 knowing exactly what's already proven,

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: arozumenko
• Source: arozumenko/sdlc-skills
• License: MIT

Install and usage instructions live in the source repository linked above.