# Drift Testing > > - **Type:** Skill - **Install:** `agentstack add skill-pactflow-pactflow-agent-skills-drift-testing` - **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior - **Seller:** [pactflow](https://agentstack.voostack.com/s/pactflow) - **Installs:** 0 - **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills) - **Latest version:** 0.1.0 - **License:** MIT - **Upstream author:** [pactflow](https://github.com/pactflow) - **Source:** https://github.com/pactflow/pactflow-agent-skills/tree/main/plugins/swagger-contract-testing/skills/drift-testing - **Website:** https://docs.pact.io/ai_tools/pactflow-skill ## Install ```sh agentstack add skill-pactflow-pactflow-agent-skills-drift-testing ``` Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent. ## About # Drift Skill Never modify the openapi spec that you are testing. ## Reference Files - [`references/test-cases.md`](references/test-cases.md) — Full test case YAML schema, all patterns, datasets, expressions - [`references/auth.md`](references/auth.md) — Authentication config, dynamic tokens, non-standard schemes, 401/403 testing - [`references/mock-server.md`](references/mock-server.md) — Local testing with Prism: setup, Prefer header, spec quality issues - [`references/lua-api.md`](references/lua-api.md) — Complete Lua API: lifecycle events, `http()`, `dbg()`, exported functions - [`references/cli-reference.md`](references/cli-reference.md) — All CLI commands/flags, configuration, parallel execution, exit codes - [`references/pactflow-and-cicd.md`](references/pactflow-and-cicd.md) — BDCT publishing workflow, GitHub Actions, GitLab CI ## Scripts - `scripts/extract_endpoints.py` — Reads the spec and outputs all operations + response codes. Summary mode flags parameters with no spec example. Scaffold mode (`--scaffold`) emits a ready-to-fill `operations:` YAML block with correct auth patterns, nil UUIDs for 404s, `ignore.schema` for 4xx, and `FILL_IN` markers. Use `--only-missing ` to generate only the gaps not yet covered by an existing test file. Requires `pyyaml`. - `scripts/check_coverage.py` — Coverage checker: diffs an OpenAPI spec against Drift test files and reports which operations and response codes are missing tests. Requires `pyyaml`. - `scripts/run_loop.sh` / `scripts/run_loop.ps1` — Feedback loop runner: retries `drift verify --failed` until all tests pass, then runs `check_coverage.py`. Both gates must pass for exit 0. Dependencies are installed automatically via uv. Use the `.ps1` version on Windows. - `scripts/start_mock.sh` / `scripts/start_mock.ps1` — Starts a Prism mock server from an OpenAPI spec. Installs Prism if needed. Supports `--port` and `--dynamic` flags. Use the `.ps1` version on Windows. Full docs: https://support.smartbear.com/swagger/contract-testing/docs/en/drift.html To discover all available pages, fetch the sitemap: `https://support.smartbear.com/swagger/contract-testing/docs/sitemap.xml` --- ## Installation ```bash # Quickest — no install needed npx @pactflow/drift --help # Project-level (recommended for teams) npm install --save-dev @pactflow/drift # Global npm install -g @pactflow/drift # Verify drift --version ``` --- ## Project Setup ```bash drift init # interactive wizard — scaffolds all files below ``` > `drift init` is interactive — ask the user to run it. ``` drift/ ├── drift.yaml # Main config — sources, plugins, global settings ├── drift.lua # Lifecycle hooks and helper functions ├── my-api.dataset.yaml # Test data └── my-api.tests.yaml # Test cases ``` Minimal `drift.yaml`: ```yaml # yaml-language-server: $schema=https://download.pactflow.io/drift/schemas/drift.testcases.v1.schema.json drift-testcase-file: v1 title: "My API Tests" sources: - name: source-oas # referenced in test targets path: ./openapi.yaml # or uri: https://... for remote specs - name: product-data path: ./product.dataset.yaml - name: functions path: ./product.lua plugins: - name: oas # spec-first verification - name: json - name: data global: auth: apply: true parameters: authentication: scheme: bearer # bearer | basic | api-key token: ${env:API_TOKEN} operations: # test cases here — see references/test-cases.md ``` --- ## Running Tests ```bash # Basic run drift verify --test-files drift.yaml --server-url https://api.example.com/v1 # Single operation (fast iteration) drift verify --test-files drift.yaml --server-url https://api.example.com/v1 --operation getProductByID # Re-run only failures drift verify --test-files drift.yaml --server-url https://api.example.com/v1 --failed # Filter by tags drift verify --test-files drift.yaml --server-url https://api.example.com/v1 --tags smoke drift verify --test-files drift.yaml --server-url https://api.example.com/v1 --tags '!destructive' ``` See `references/cli-reference.md` for all flags, parallel execution, JUnit output, and exit codes. --- ## Full Coverage Feedback Loop When the goal is full endpoint coverage: > **Caution — destructive tests on production:** If `--server-url` points at a live > production API, DELETE and POST tests are permanent. Always use a dedicated test account > and confirm any resource used in a DELETE test is disposable. Copy this checklist and track your progress: ``` Coverage Loop Progress: - [ ] Step 0: Check current coverage (check_coverage.py) - [ ] Step 1: Parse spec and collect operation list (openapi-parser skill or extract_endpoints.py) - [ ] Step 2: Assemble initial test file - [ ] Step 3: Run tests (run_loop.sh / run_loop.ps1) - [ ] Step 4: Diagnose and fix each failure - [ ] Step 5: Apply common fixes (hooks for state, data seeding) - [ ] Step 6: Verify exit code 0 + full coverage ``` ### Step 0 — Check current coverage Run before writing tests or when resuming an existing test suite: ```bash # Run against your spec and test file(s) uv run path/to/scripts/check_coverage.py \ --spec openapi.yaml \ --test-files drift.yaml # Multiple files / globs uv run path/to/scripts/check_coverage.py \ --spec openapi.yaml \ --test-files "tests/*.yaml" # Machine-readable output (for CI or scripting) uv run path/to/scripts/check_coverage.py \ --spec openapi.yaml \ --test-files drift.yaml --json ``` Output shows: operations with no tests at all, operations missing specific response codes, and overall operation/code percentages. Exit code 0 = full coverage, 1 = gaps remain. The script excludes 500/501/502/503 by default (same rule as Step 1 below). Pass `--exclude-codes` to customise. ### Step 1 — Parse the spec Use `extract_endpoints.py` to collect the complete operation list, all documented response codes per operation, and ready-to-use `operations:` YAML stubs. If the **openapi-parser skill** is available in your environment, you can use that instead. ```bash # See all operations + response codes, flagging params with no spec example uv run scripts/extract_endpoints.py --spec openapi.yaml # Generate skeleton stubs for every operation uv run scripts/extract_endpoints.py --spec openapi.yaml \ --scaffold --source my-oas > operations.yaml # Generate ONLY the gaps not already in an existing test file uv run scripts/extract_endpoints.py --spec openapi.yaml \ --scaffold --only-missing drift.yaml --source my-oas >> drift.yaml ``` ``` GET /products → 200, 401, 404 POST /products → 201, 400, 401 DELETE /products/{id} → 204, 401, 403, 404 ``` **Critical:** Any parameter without a spec-level `example` causes `Value for query parameter X is missing`. Supply an explicit value in `parameters.query/path/headers` for each. **Globally-required query parameters** (e.g. `?version=YYYY-MM-DD` on every endpoint) can be injected once via the `http:request` hook rather than repeated in every test case: ```lua ["http:request"] = function(event, data) if data.query == nil then data.query = {} end data.query["version"] = "2024-01-04" return data -- MUST return modified data end ``` **Duplicate `operationId` values** — some specs reuse the same operationId for two different paths. Use `method:path` targeting for the affected operation: ```yaml target: source-oas:post:/orgs/{org_id}/apps/installs/{install_id}/secrets ``` **500 responses are excluded from the coverage requirement** — a 500 requires a server bug and can't be deterministically triggered. ### Step 2 — Assemble the initial test file Wire the stubs from the openapi-parser into `drift.yaml`. Don't aim for perfection — the loop surfaces what's missing. Start each test as simple as possible: ```yaml getProduct_Success: target: source-oas:getProductByID parameters: path: id: 10 expected: response: statusCode: 200 ``` Add tags to every operation — they enable `--tags` filtering and make suites easier to manage: ```yaml getProduct_Success: target: source-oas:getProductByID tags: [smoke, read-only, products] ... getProduct_Unauthorized: tags: [security, auth] ... deleteProduct_Success: tags: [destructive, products] ... ``` Common tags: `smoke`, `read-only`, `write`, `destructive`, `security`, `auth`, `regression`. See `references/test-cases.md` for the full tags section. For error paths, see `references/test-cases.md` for 401, 403, 404, and 400 patterns. For mock server setup, see `references/mock-server.md`. ### Step 3 — Run and capture failures The `run_loop.sh` script automates this entire step through Step 6: ```bash # Runs drift --failed in a loop, then checks coverage. Exits 0 only when both pass. path/to/scripts/run_loop.sh \ --spec openapi.yaml \ --test-files drift.yaml \ --server-url https://api.example.com/v1 ``` Or run drift manually and iterate: ```bash drift verify --test-files drift.yaml --server-url https://api.example.com/v1 # Re-run only failures to keep the loop fast drift verify --test-files drift.yaml --server-url https://api.example.com/v1 --failed ``` For local testing with a mock server, start Prism first: ```bash path/to/scripts/start_mock.sh --spec openapi.yaml --port 4010 # then in another terminal: path/to/scripts/run_loop.sh --spec openapi.yaml --test-files drift.yaml --server-url http://localhost:4010 ``` ### Step 4 — Diagnose and fix each failure | Symptom | Likely cause | Fix | | ---------------------------------------- | --------------------------------------------------- | ----------------------------------------------------------------------------------- | | Got 404, expected 200 | Test data doesn't exist | Add `operation:started` hook to seed the resource | | Got 200, expected 404 | ID happens to exist | Use `${notIn(...)}` or nil UUID `00000000-0000-0000-0000-000000000000` | | Got 401, expected 200 | Auth not configured | Add `global.auth` or check token env var | | Got 200, expected 401 | Auth not stripped | Add `exclude: [auth]` + bad token | | Got 403, expected 200 | Token lacks required scope | Use a token with sufficient permissions | | Got 200, expected 403 | Need valid auth + forbidden resource | Point at a resource the token can't access; see `references/auth.md` | | Schema validation error on response | API drifted from spec, OR spec has invalid examples | Check whether spec examples are valid — Drift may be correctly reporting a spec bug | | `Value for query parameter X is missing` | Optional param has no spec `example` | Supply an explicit value for every param without a spec example | | Got 400 on a 200 test | Missing globally-required query param | Inject it via `http:request` hook or add to every test case | | Got 500 | Test data triggered a server bug | Fix the data | **`ignore: { schema: true }`** suppresses request schema validation only. Use it on any 4xx scenario — especially when testing against a mock server, where Prism doesn't enforce auth and may return an inaccurate error body. Response schema validation has no bypass; spec example bugs surface as failures (see `references/mock-server.md`). **Multiple 2xx codes:** Write one test per documented code — `statusCode: [200, 204]` array syntax is not supported. **Dynamic IDs and hook timing:** Dataset expressions resolve _before_ `operation:started`. Use pre-seeded static IDs, or rewrite the URL via `http:request`. ### Step 5 — Common fixes **Data must exist before the test (DELETE, PUT, PATCH):** ```lua ["operation:started"] = function(event, data) if data.operation == "deleteProduct_Success" then http({ url = server_url .. "/products", method = "POST", body = { id = 10, name = "test", price = 9.99 } }) end end, ["operation:finished"] = function(event, data) http({ url = server_url .. "/products/10", method = "DELETE" }) end, ``` See `references/lua-api.md` for the full Lua API and the `data` object shape. ### Step 6 — Verify exit code 0 + full coverage ```bash drift verify --test-files drift.yaml --server-url https://api.example.com/v1 echo "Exit code: $?" ``` Before declaring done, verify coverage is complete: ```bash uv run path/to/scripts/check_coverage.py \ --spec openapi.yaml --test-files drift.yaml echo "Coverage exit: $?" ``` Done when both commands exit 0. --- ## Quick Reference | Scenario | Approach | | ---------------------------------- | ---------------------------------------------- | | Stateless read-only endpoint | Declarative test, no hooks | | Stable test data | Dataset expressions | | Create data before test | `operation:started` hook | | Clean up after test | `operation:finished` hook | | Dynamic values (UUIDs, timestamps) | `exported_functions` in Lua | | Guaranteed 404 | `${notIn(...)}` or nil UUID | | Force error code on mock server | `Prefer: code=X` header | | Test without live backend | Prism mock — see `references/mock-server.md` | | Non-standard auth prefix | `http:request` hook — see `references/auth.md` | | Re-run only broken tests | `--failed` flag | | Publish to PactFlow | `--generate-result` flag | ## Source & license This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code. - **Author:** [pactflow](https://github.com/pactflow) - **Source:** [pactflow/pactflow-agent-skills](https://github.com/pactflow/pactflow-agent-skills) - **License:** MIT - **Homepage:** https://docs.pact.io/ai_tools/pactflow-skill Install and usage instructions live in the source repository linked above. ## Pricing - **Free** — Free ## Security capabilities Automated source analysis of v0.1.0 — what this tool can access: - **Network access:** no - **Filesystem access:** no - **Shell / process execution:** no - **Environment & secrets:** no - **Dynamic code execution:** no *"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.* ## Versions - **0.1.0** — security scan: passed — Imported from the upstream source. ## Links - Listing page: https://agentstack.voostack.com/l/skill-pactflow-pactflow-agent-skills-drift-testing - Seller: https://agentstack.voostack.com/s/pactflow - Browse the marketplace: https://agentstack.voostack.com/browse --- Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.