Check Tests

check-tests: Test Suite Runner & Coverage Tracker

Runs the configured test command, tracks pass/fail counts and coverage percentage over time, and alerts when regressions appear or coverage drops. READ-ONLY in terms of PRs — writes only to agent/state/test_coverage.json.

---

Safety Rules

1. HALT File — Check config.safety.halt_file first. If it exists, print reason and stop.
2. Read-only — Writes only to agent/state/test_coverage.json. Never touches test files or source.
3. No test_command — If config.test_command is empty or unset, skip gracefully.

---

Step 0: Safety

0-A: HALT Check

if file exists at config.safety.halt_file:
  Print "[HALT] check-tests stopped. Reason: {file_content}"
  EXIT immediately.

0-B: Config Validation

if config.test_command is missing or empty:
  Print "No test command configured. Skipping check-tests."
  Print "Set config.test_command (e.g. \"npm test\", \"pytest\", \"go test ./...\") to enable."
  EXIT cleanly (not an error).

---

Step 1: Load Previous State

Read agent/state/test_coverage.json. If missing, initialize with: { "schema_version": "1.0.0", "last_run": null, "run_count": 0, "status": "unknown", "results": { "total": 0, "passed": 0, "failed": 0, "skipped": 0, "coverage_pct": null }, "previous_results": null, "alerts": [], "history": [] }

Note previous passed, failed, and coverage_pct values for Step 3.

---

Step 2: Run Tests

Execute with a configurable timeout (read config.test_timeout_seconds; default 300):

{config.test_command} 2>&1

Capture exit code, stdout, stderr. Parse:

• Counts: try framework-specific patterns in order:
• Jest: Parse each token independently from the Tests: summary line — (\d+) failed, (\d+) passed, (\d+) skipped, (\d+) total as separate optional matches (default 0 when absent). IMPORTANT: the Tests: line omits "failed" when all tests pass (e.g., Tests: 26 passed, 26 total), so a combined pattern requiring all tokens will fail.
• pytest: (\d+) passed, (\d+) failed, (\d+) skipped, (\d+) error
• Go: count lines matching ^ok\s+\t as passed packages, lines matching ^FAIL\t as failed packages (IMPORTANT: Go emits multiple lines containing FAIL per failed package — --- FAIL: TestName, standalone FAIL, FAIL\tpkg/path, and a trailing FAIL summary. ONLY ^FAIL\t followed by a package path represents a failed package. Similarly, --- PASS: lines are per-test, not per-package.) In verbose mode (-v), also count --- PASS: lines for individual test counts and --- FAIL: for individual test failures, reporting both: Tests: {test_passed}/{test_total} passed (packages: {pkg_passed}/{pkg_total})
• Mocha: (\d+) passing, (\d+) failing, (\d+) pending
• RSpec: (\d+) examples?,\s*(\d+) failures?(?:,\s*(\d+) pending)?
• Rust/Cargo: test result: (?:ok|FAILED)\.\s*(\d+) passed;\s*(\d+) failed;\s*(\d+) ignored — Cargo emits a single summary line. ignored maps to skipped. Coverage is not emitted by default; requires cargo-tarpaulin or cargo llvm-cov.
• Python unittest (stdlib python -m unittest): Ran\s+(\d+)\s+tests? gives total. If a line matches ^OK\b → failed=0, passed=total (OK \(skipped=(\d+)\) sets skipped). If a line matches ^FAILED\s*\((.+)\) → inside the parens read failures=(\d+) and errors=(\d+) (sum → failed) and skipped=(\d+) if present; then passed = total - failed - skipped. unittest emits NO coverage — use python3 -m coverage run -m unittest && coverage report for the pytest-cov TOTAL ... % line. (Listed — and therefore tried — BEFORE Bun, since Bun also matches Ran (\d+) tests.)
• Bun: (\d+)\s+pass(?:\b) for passed, (\d+)\s+fail(?:\b) for failed — Bun uses present tense (pass/fail) NOT past tense (passed/failed). Also Ran\s+(\d+)\s+tests for total. (\d+)\s+skip for skipped. Coverage requires --coverage flag.
• Vitest: Uses the same Istanbul/v8 table format as Jest for coverage. Test counts use Tests\s+(\d+)\s+passed\s+\((\d+)\) format — note: no colon after Tests, no comma separators. Parse each token independently as with Jest.
• Fallback: generic (\d+)\s+(?:tests?\s+)?passed, (\d+)\s+(?:tests?\s+)?failed
• Go skipped: count --- SKIP: lines for skipped tests (only visible in verbose -v mode; if not verbose, skipped count defaults to 0)
• Compute total = passed + failed + skipped when the framework does not emit a total
• Coverage: try patterns in order:

1. All files\s*\|\s*([\d.]+) (Istanbul/nyc table format — NOTE: data rows use bare numbers, no % sign. The first column after All files | is statement coverage.)
2. TOTAL\s+.*?([\d.]+)% (pytest-cov)
3. coverage:\s*([\d.]+)% (Go)
4. Statements\s*:\s*([\d.]+)% (Jest text-summary reporter, NOT the default table)
5. ([\d.]+)%\s*coverage (generic fallback)

Use first match; if none match record null. Go multi-package note: Go emits one coverage: line per package. When multiple matches exist, compute the average across all matched values (this approximates aggregate coverage since Go does not produce a single aggregate figure). Ignore coverage: 0.0% from packages with [no test files].

• Status: exit 0 → "passing", exit 127 (command not found) or 126 (permission denied) → "error" with detail "test command not found or not executable", timeout → "error" with detail "timeout after Ns", other non-zero → "failing"

---

Step 3: Detect Regressions

Skip regression detection when: (a) first run (no previous state), (b) current run status is "error", or (c) previous run status was "error". In these cases record results only, no alerts. Otherwise compare against the last successful ("passing" or "failing") run:

| Condition | Type | Severity | |-----------|------|----------| | failed increased by > 5 | regression | critical | | failed increased by 1–5 | regression | warning | | coverage dropped by > 5% | coverage_drop | warning | | failed → 0 (was > 0) | recovery | info |

Alert format: {"severity": "warning", "type": "regression", "detail": "3 new failures (was 2, now 5)"}

---

Step 4: State Update

Write to agent/state/test_coverage.json:

{
  "schema_version": "1.0.0",
  "last_run": "ISO 8601",
  "run_count": N,
  "status": "passing|failing|error",
  "results": { "total": N, "passed": N, "failed": N, "skipped": N, "coverage_pct": N.N },
  "previous_results": { "...previous results object..." },
  "previous_status": "passing|failing|error",
  "new_failures": N,
  "coverage_drop": N.N,
  "alerts": [{"severity": "warning", "type": "regression", "detail": "..."}],
  "history": [{"timestamp": "...", "passed": N, "failed": N, "coverage_pct": N.N}]
}

• previous_status — the prior run's status, persisted so Step 3's rule (c)

("previous run status was error") is actually decidable next run.

• new_failures — max(0, results.failed - previous_results.failed) this run

(0 on first run). coverage_drop — max(0, previous coverage_pct - current) in percentage points. These two are the variables evolve's 4-B chain trigger evaluates (new_failures >= 1 OR coverage_drop > 5) — they must be written EVERY run, not only when an alert fires.

History: append the current run, then truncate to the most recent 50 entries (drop oldest first). If the array already exceeds 50 (e.g., manual edits), truncate to 50 in this write.

---

Step 5: Report

Tests: {passed}/{total} passed, {skipped} skipped {coverage_section}
Status: {passing|failing|error}
vs Previous: {delta_section or "first run / no comparison available"}
Alerts: {alert list or "none"}

Where:

• {coverage_section} = (X.X% coverage) when available, or (coverage: n/a) when coverage_pct is null.
• If total is 0 and status is not "error", display Tests: 0 found (check test_command output).
• {delta_section} omits coverage delta when either the current or previous coverage_pct is null.

Example: Tests: 142/145 passed, 0 skipped (87.3% coverage) | Status: failing | vs Previous: +3 failed, coverage -1.2% | Alerts: [warning] regression — 3 new failures

---

Graceful Degradation

• No test_command → skip with message
• Test command fails to start → record "error" status, write state
• Coverage parsing fails → record coverage_pct: null, continue
• Timeout (exceeds config.test_timeout_seconds, default 300) → kill process, record "error" with timeout note
• State file corrupt → treat as first run, re-initialize

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: mataeil
• Source: mataeil/OODA-loop
• License: MIT

Install and usage instructions live in the source repository linked above.