Incident Retro Workflow

Incident Retro Workflow

Overview

Incident retros turn painful evidence into safer systems. Build a sourced timeline, quantify impact, separate triggers from factors, and create verified actions that reduce recurrence, detection time, or response time.

When to Use

| Use When | Don't Use When | |----------|----------------| | The user says "incident retro", "postmortem", "outage review", "rollback review", "RCA", "timeline", or "corrective actions" | The incident is still active and needs mitigation | | An incident, customer-impacting bug, deploy failure, data issue, or reliability regression needs analysis | A bug has no reproduction or root-cause evidence; use systematic-debugging | | Alerts, logs, tickets, deploys, traces, metrics, or chat need to become a factual retro | No artifact, timeline, symptom, or affected behavior exists | | Follow-up work needs owners, dates, verification, and risk reduction | The task only adds telemetry; use observability-workflow | | A draft retro needs review for blame, unsupported claims, weak actions, or missing impact | Security, privacy, tenant, abuse, or legal exposure lacks review |

Phase 0 Benchmark Decision

Type: Technique with safety judgment. Use blameless habits: facts before interpretation, explicit uncertainty, impact before causes, people-neutral language, and evidence-tied actions.

Retro Gate

Incident active -> no final retro.
No evidence or timeline -> no root-cause claim.
Person-blame language -> rewrite to system/process factors.
No owner/date/verification -> invalid action item.
Security/privacy/legal exposure -> use agent-security-guard.

The Process

stabilize -> evidence -> timeline -> impact -> factors -> actions -> review -> handoff

Step 1: Confirm Stabilization, Scope, and Retro Mode

Expected Output: status=, incident=, severity=, scope=, retro_mode=

Confirm whether the system is restored, mitigated, or still failing. Name the incident, window, affected systems, audience, and draft/final mode.

git status --short
rg -n "incident|outage|rollback|postmortem|retro|SEV|severity|customer impact|data loss|deploy|runbook|mitigation|resolved" .

Record:

Incident:
Status:
Start/end:
Affected systems/users:
Retro mode:

If fails:

• status=ongoing -> stop final retro work; produce a containment note.
• Scope is unclear -> ask for start time, end time, affected system, and user impact.
• Security, privacy, tenant, abuse, legal, or personnel-sensitive facts appear -> use agent-security-guard.

Step 2: Collect Evidence and Label Confidence

Expected Output: evidence=, gaps=, confidence=

Collect primary artifacts first: alerts, metrics, traces, logs, deploy records, channel timestamps, tickets, status updates, runbooks, commits, and customer reports.

rg -n "error|timeout|exception|rollback|deploy|migration|alert|SLO|SLA|health|metric|trace|log|status page|customer report" .
git log --oneline --decorate -n 30
git diff --name-status

Evidence table:

| Evidence | Source | Timezone | Proves | Confidence |
|----------|--------|----------|--------|------------|

If fails:

• Only anecdotes exist -> mark confidence=low; do not claim root cause.
• Evidence conflicts -> keep both versions with source and timestamp.
• Artifacts contain secrets, PII, customer data, or personnel data -> redact before quoting.

Step 3: Build a Factual Timeline

Expected Output: timeline=, detection=, mitigation=, resolution=, unknowns=

Use absolute timestamps with timezone. Separate detection, acknowledgement, diagnosis, mitigation, recovery, customer communication, and post-recovery checks.

| Time | Event | Source | Confidence |
|------|-------|--------|------------|

Rules:

• Include detected, acknowledged, mitigation started, mitigated, resolved, and verified.
• Label inference as inference.
• Keep unknown time ranges visible.

If fails:

• Times lack timezone -> ask or label timezone as unknown.
• Events are out of order -> sort by absolute time and preserve source timestamps.
• Detection or resolution is missing -> add an open question instead of smoothing the narrative.

Step 4: Quantify Impact and Blast Radius

Expected Output: impact=, duration=, severity=, blast_radius=

Quantify user harm, data correctness, availability, latency, toil, support load, SLA/SLO effect, and communications. Do not invent numbers.

| Dimension | Value | Evidence | Notes |
|-----------|-------|----------|-------|
| Users affected |  |  |  |
| Duration |  |  |  |
| Data impact |  |  |  |
| Business/SLO impact |  |  |  |

If fails:

• Impact is unknown -> state the missing artifact that would prove it.
• Severity label conflicts with measured impact -> use measured impact and note mismatch.
• Data loss, privacy, payment, auth, tenant, or destructive action appears -> use agent-security-guard.

Step 5: Identify Triggers and Contributing Factors

Expected Output: factors=, root_cause_claim=

Prefer contributing factors over a single root cause unless evidence proves one dominant cause.

| Category | Question | |----------|----------| | Trigger | What changed or happened immediately before impact? | | Latent condition | What existing weakness made the trigger harmful? | | Guardrail gap | What test, review, deploy gate, flag, or rollback missed it? | | Detection gap | Why was it detected then, not earlier? | | Response gap | What slowed mitigation, communication, or recovery? | | Ownership gap | What unclear owner, doc, or decision path mattered? |

| Factor | Category | Evidence | Link to impact | Confidence |
|--------|----------|----------|----------------|------------|

If fails:

• Explanation says "human error" -> rewrite as the system/process condition that allowed the action.
• Cause is asserted without evidence -> demote to hypothesis and add an open question.
• Factors point to missing tests or signals -> use test-design-workflow or observability-workflow.

Step 6: Create Corrective Actions with Verification

Expected Output: actions=, prevention=, detection=, response=, residual_risk=

Every action must reduce recurrence, detection time, impact, or response time. Each needs owner, due date, linked factor, and closure proof.

| ID | Action | Type | Owner | Due | Verify | Linked factor |
|----|--------|------|-------|-----|--------|---------------|

Action rules:

• Use concrete verbs: add, remove, block, alert, test, document, migrate, rehearse, automate.
• Verify with a test, alert evidence, dashboard query, runbook link, deploy check, review evidence, or dry run.
• Track residual risk when action is deferred.

If fails:

• Action has no owner, due date, or verification -> mark invalid and revise.
• Action says "improve monitoring", "add tests", or "document better" -> convert to exact signal, case, file, runbook, or gate.
• Action spans teams, releases, or risky sequencing -> use planning-workflow.

Step 7: Review the Retro for Quality and Safety

Expected Output: review=, gaps=, safety=, confidence=

Review before publication. The draft must be factual, blameless, specific, and actionable.

rg -n "blame|fault|careless|negligent|human error|TBD|TODO|unknown|probably|obvious|just add|monitoring later|add tests" .

Checks:

• Timeline has sources, timestamps, timezone, and unknowns.
• Impact is quantified or explicitly unknown.
• Trigger, symptom, factors, and root-cause claims are separated.
• People are not blame anchors.
• Actions link to factors and have owner, due date, and verification.
• Sensitive data is redacted; open questions are separate from conclusions.

If fails:

• Blame language remains -> rewrite around system condition, decision context, guardrail, or process gap.
• Unsupported certainty remains -> lower confidence or add evidence.
• Sensitive facts remain -> redact or route through agent-security-guard.

Step 8: Handoff Follow-up and Learning Loop

Expected Output: status=, retro=, follow_up=, next_skill=

Finish with a publishable retro and trackable follow-up.

Incident:
Status:
Impact:
Timeline:
Contributing factors:
What went well:
What went poorly:
Corrective actions:
Open questions:
Residual risk:
Follow-up:
Status: ready=

| State | Meaning | |-------|---------| | ready | Evidence, timeline, impact, factors, actions, and safety review are complete | | with-gaps | Unknowns or residual risks are explicit and non-blocking | | blocked | Missing evidence, active incident, unsafe disclosure, or invalid actions blocks publication |

If fails:

• Follow-up is not trackable -> add owners, dates, and verification artifacts.
• ready has open blockers -> downgrade to with-gaps or blocked.
• Actions need implementation sequencing -> use planning-workflow.

Bad/Good Retro Patterns

Pattern 1: Person Blame

Bad: The deployer forgot to update config, causing the outage.

Good: The deploy path allowed config/code drift; no preflight check blocked the incompatible pair.

Pattern 2: Unsupported Root Cause

Bad: Root cause: database overload.

Good: Contributing factor: checkout queries exceeded the pool after migration increased per-request queries from 2 to 9.

Pattern 3: Weak Action Item

Bad: Add monitoring and more tests.

Good: A2: Add checkout 5xx ratio alert, owner=Payments, due=2026-07-15, verify=staging failure injection plus alert evidence.

Rationalization Table

| Rationalization | Why it fails | |-----------------|--------------| | "We already know what happened" | Memory loses timestamps, impact, uncertainty, and context. | | "It was human error" | Human error names a symptom; prevention needs system factors. | | "No customer complained" | Lack of reports is not proof of no impact. | | "Just add monitoring" | Monitoring without owner, threshold, runbook, and test is noise. | | "Action items can be broad" | Broad actions do not close because verification is undefined. |

Red Flags

• You write the story before collecting artifacts.
• Timeline events lack timezone, source, or confidence.
• Impact says "minimal" or "no customer impact" without evidence.
• Root cause is a person, team, deploy, or tool with no latent factor.
• Actions lack owner, due date, verification, or linked factor.
• Actions target prevention but ignore detection, mitigation, and response.
• Open questions are hidden inside conclusions.
• Sensitive customer, security, personnel, or legal facts appear in the draft.

Common Pitfalls

| Symptom | Root cause | Fix | |---------|------------|-----| | Retro reads like a story | Narrative was written before artifacts | Build the evidence table and timeline first | | Team argues about root cause | Trigger, latent condition, and factors were collapsed | Split factors by category and evidence | | Action items never close | Owner, due date, and verification were omitted | Reject actions without all three | | Same incident repeats | Actions targeted symptoms, not guardrails | Link each action to a factor and risk reduction | | Blame hides useful facts | Person-centered language makes disclosure unsafe | Rewrite around system condition and missing guardrail | | No-impact claim is disputed | Impact was not reconciled | Compare metrics, tickets, status updates, and reports |

Verification Checklist

• [ ] Description triggers on retros, postmortems, outage reviews, rollback reviews, timelines, RCA, and corrective actions.
• [ ] Incident status, scope, severity, time window, and retro mode are explicit.
• [ ] Evidence lists sources, timestamps/timezone, facts proved, gaps, and confidence.
• [ ] Timeline includes detection, acknowledgement, mitigation, resolution, verification, and unknowns.
• [ ] Impact quantifies users/systems/data/business/SLO or states exactly what is unknown.
• [ ] Symptoms, trigger, factors, and root-cause claims are separated.
• [ ] Language is blameless and avoids person-centered blame.
• [ ] Actions have owner, due date, verification, linked factor, and action type.
• [ ] Sensitive information is redacted or routed through agent-security-guard.
• [ ] Final status is ready, with-gaps, or blocked with residual risk named.

Interaction with Other Skills

| Related Skill | Trigger | Handoff | |---------------|---------|---------| | systematic-debugging | Root cause is unknown or speculative | Reproduce and isolate before final claims | | observability-workflow | Missing signals or runbooks contributed | Convert detection/triage gaps into signal design | | planning-workflow | Actions span teams, releases, or risky sequencing | Turn actions into phases, gates, rollback, and owners | | test-design-workflow | Missing tests or weak regression coverage contributed | Design success, failure, boundary, and regression tests | | code-review-workflow | Review missed the failure or follow-up diff needs review | Audit guardrails, tests, rollback, and signals | | agent-security-guard | Security, privacy, tenant, payment, secrets, legal, or destructive boundaries appear | Redact and safety-review before publishing | | cross-model-verification | Findings are high-risk, disputed, or low-confidence | Review missing factors and weak actions | | requirements-clarifier | Scope, impact, ownership, or closure criteria are vague | Clarify outcome and acceptance criteria |

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: 2702207741-dev
• Source: 2702207741-dev/agent-skills-pipeline
• License: MIT
• Homepage: https://github.com/2702207741-dev/agent-skills-pipeline/blob/main/docs/README.md

Install and usage instructions live in the source repository linked above.