Regression Sweep

Regression Sweep

Walk the entire validated/*.json tree, re-fire each finding's poc.py, compare output against the recorded poc_output.txt, and write a weekly drift report. Mounted onto the cloud-agent task #4.

Trigger

Cron weekly (default Mondays 02:00 UTC). May also be invoked ad-hoc after a major patch deployment.

Workflow

1. Index validated findings. Glob validated/*.json and resolve each entry's FINDING_DIR (under findings/finding-NNN/).
2. Per finding:

• Re-run python3 poc.py with a 60-second timeout.
• Capture stdout/stderr into findings/finding-NNN/evidence/validation/regression-{week}-rerun.txt.
• Diff against findings/finding-NNN/evidence/validation/poc-rerun-output.txt (the validator's original re-run output) using a normalized line-set comparison (strip timestamps, request IDs, ephemeral tokens).
• Re-check the finding's CVE via tools/nvd-lookup.py — has severity changed?

1. Classify each finding into one of:

• still_valid — re-run matches baseline within tolerance, CVSS unchanged.
• drift_severity — re-run matches, but CVSS shifted ≥1.0 (NVD re-scored).
• newly_invalid — re-run output diverges, exploit no longer fires. Likely patched.
• newly_revalidated — finding had been marked REJECTED later, but now fires again. Regression.
• inconclusive — re-run errored (network, target unreachable). Retry next sweep.

1. Write report to artifacts/regression-{YYYYWww}.json + human-readable regression-{YYYYWww}.md. Transitions (newly_revalidated, drift_severity, newly_invalid) appear in the report under explicit headers for analyst review.

Output

{OUTPUT_DIR}/
  artifacts/
    regression-{YYYYWww}.json         # machine-readable result
    regression-{YYYYWww}.md           # human summary
  findings/finding-NNN/evidence/validation/
    regression-{YYYYWww}-rerun.txt    # captured re-run output

regression-{week}.json schema:

{
  "week": "2026W19",
  "swept_at": "2026-05-13T02:00:00Z",
  "counts": {"still_valid": 47, "drift_severity": 2, "newly_invalid": 5, "newly_revalidated": 1, "inconclusive": 3},
  "findings": [
    {"finding_id": "finding-012", "asset": "asset42", "cve": "CVE-2024-12345",
     "verdict": "newly_invalid", "reason": "PoC output diverged: response now 404",
     "baseline_cvss": 9.8, "current_cvss": 9.8}
  ]
}

Rules

1. Demonstrate, never disrupt. Re-firing a PoC is observation, not mutation. Every poc.py already satisfied the demonstrate-only constraints at validation time (task-03 Safety section); the sweep simply re-executes the same script — which by contract reads a proof signal and exits. If a PoC at re-run attempts a mutating action it should not have contained originally, the sweep aborts that finding with inconclusive and emits a stderr WARN — the PoC needs re-validation, not regression scoring.
2. Bounded per-PoC time. 60-second timeout per poc.py. Timeouts → inconclusive, not newly_invalid. Avoids false-positive "patched" claims caused by network blips.
3. Normalized diff. Strip timestamps (\d{4}-\d{2}-\d{2}T\d{2}:\d{2}), request-IDs ([a-f0-9]{32,}), and ephemeral session tokens before comparing. Real exploit output is structurally stable.
4. No new findings. A regression sweep can flip status of existing findings but cannot create new ones. Newly observed vulns belong to the Validation Run task, not this skill.
5. Idempotent. Re-running the same week's sweep overwrites the same regression-{YYYYWww}.json. The per-finding regression-{week}-rerun.txt is timestamped to preserve history.
6. Cap concurrent re-runs. Max 5 parallel PoC re-runs per sweep to avoid hammering production assets.

References

• reference/diff-normalization.md — full normalization rules and per-finding output-stability heuristics.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: transilienceai
• Source: transilienceai/communitytools
• License: MIT
• Homepage: https://www.transilience.ai/

Install and usage instructions live in the source repository linked above.