# Incident Postmortem > Write blameless incident postmortems with structured timeline reconstruction, impact quantification, contributing factor analysis, and actionable follow-up items with owners and deadlines. - **Type:** Skill - **Install:** `agentstack add skill-the-ai-directory-company-agents-and-skills-incident-postmortem` - **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior - **Seller:** [The-AI-Directory-Company](https://agentstack.voostack.com/s/the-ai-directory-company) - **Installs:** 0 - **Category:** [AI & ML](https://agentstack.voostack.com/c/ai-and-ml) - **Latest version:** 0.1.0 - **License:** MIT - **Upstream author:** [The-AI-Directory-Company](https://github.com/The-AI-Directory-Company) - **Source:** https://github.com/The-AI-Directory-Company/agents-and-skills/tree/main/skills/incident-postmortem - **Website:** https://ai-directory.company ## Install ```sh agentstack add skill-the-ai-directory-company-agents-and-skills-incident-postmortem ``` Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent. ## About # Incident Postmortem ## Before you start Gather the following from the user: 1. **What happened?** (Service name, symptoms, error messages, alerts that fired) 2. **When did it happen?** (Detection time, start time if known, resolution time — all in UTC) 3. **Who was involved?** (On-call responder, escalation chain, any external parties) 4. **What was the blast radius?** (Affected users, regions, services, revenue impact) 5. **What fixed it?** (Mitigation steps taken, in order) If the user gives you a vague summary ("the site went down for a bit"), push back: "What specific errors did users see? Which services were affected? When exactly did alerts fire vs. when was the issue resolved?" ## Postmortem template Use the following structure for every postmortem: ### Incident Summary Write 3-5 sentences covering: what broke, who was affected, how long it lasted, and how it was resolved. This should be understandable by someone outside the team. ``` On 2024-03-12 at 14:32 UTC, the checkout service began returning 500 errors for all payment processing requests. Approximately 12,000 users were unable to complete purchases during the 47-minute outage. The issue was caused by an expired TLS certificate on the payment gateway. Service was restored at 15:19 UTC by rotating the certificate. ``` ### Timeline Use UTC timestamps. Include detection lag (time between incident start and first alert). Mark each entry with a category tag. ``` 14:32 UTC [ONSET] First 500 errors appear in payment service logs 14:38 UTC [DETECTION] PagerDuty alert fires for checkout error rate > 5% 14:40 UTC [RESPONSE] On-call engineer acknowledges alert 14:45 UTC [DIAGNOSIS] Engineer identifies TLS handshake failures in logs 14:52 UTC [ESCALATION] Platform team paged for certificate access 15:10 UTC [MITIGATION] New certificate issued and deployed to staging 15:15 UTC [MITIGATION] Certificate deployed to production 15:19 UTC [RESOLUTION] Error rates return to baseline, incident closed ``` ### Impact Quantify impact with actual numbers, not vague language: - **Duration**: Total outage time (onset to resolution) and user-facing downtime - **Users affected**: Count or percentage, segmented if possible - **Revenue impact**: Lost transactions, failed payments, SLA credits issued - **Downstream effects**: Other services or teams that were impacted - **Detection time**: How long between onset and first alert ### Contributing Factors List every factor that contributed to the incident occurring or lasting longer than it should have. Frame these as system failures, not personal failures. ``` - Certificate expiry was tracked in a spreadsheet with no automated alerting - The payment service had no fallback path when TLS negotiation fails - Runbook for certificate rotation was last updated 18 months ago and referenced a deprecated tool - On-call engineer did not have permissions to rotate certificates, requiring escalation ``` ### Root Cause Identify the deepest systemic cause. The root cause is never "someone made a mistake" — it is the system condition that allowed the mistake to have impact. ``` Root cause: Certificate lifecycle management relied on manual tracking without automated expiry alerts or rotation. The system had no defense against expiry because it was treated as a one-time setup rather than an ongoing concern. ``` ### Action Items Every action item must have an owner, a deadline, and a priority. Use this format: | Priority | Action Item | Owner | Deadline | Ticket | |----------|-------------|-------|----------|--------| | **P0** | Add automated certificate expiry alerting (30/14/7 day warnings) | @platform-team | 2024-03-19 | OPS-891 | | **P1** | Implement certificate auto-rotation for payment service | @platform-team | 2024-04-01 | OPS-892 | | **P1** | Grant on-call engineers certificate rotation permissions | @security-team | 2024-03-15 | SEC-234 | | **P2** | Add TLS handshake failure to checkout service health check | @checkout-team | 2024-04-15 | CHK-567 | Priority definitions: **P0** — before next on-call rotation, prevents recurrence. **P1** — within 2 weeks, reduces severity or detection time. **P2** — within 30 days, improves resilience or observability. ### Lessons Learned Include three categories: - **What went well**: Response actions, tools, or processes that worked as intended - **What went poorly**: Gaps that made the incident worse or slower to resolve - **Where we got lucky**: Things that could have made this much worse but didn't ## Quality checklist Before delivering the postmortem, verify: - [ ] Summary is understandable by someone outside the engineering team - [ ] Timeline uses UTC and includes detection lag - [ ] Impact section contains actual numbers, not "some users were affected" - [ ] Contributing factors describe system failures, not individual mistakes - [ ] Root cause identifies a systemic issue, not "human error" - [ ] Every action item has an owner, deadline, priority, and ticket reference - [ ] At least one P0 action item exists that prevents immediate recurrence - [ ] Lessons learned include all three categories (well, poorly, lucky) ## Common mistakes to avoid - **Blaming individuals**. "John forgot to renew the certificate" is a blame statement. "Certificate renewal depended on manual tracking with no automated alerts" is a system observation. Always describe the system gap, not the person. - **Vague action items**. "Improve monitoring" is not actionable. "Add PagerDuty alert when certificate expiry is within 30 days (OPS-891, @platform-team, due 2024-03-19)" is actionable. - **Missing the detection gap**. Always call out how long the incident was occurring before anyone noticed. A 2-minute outage with a 45-minute detection gap is a monitoring problem, not just an infrastructure problem. - **Action items without owners**. An action item assigned to a team mailing list or "TBD" will not get done. Every item needs a specific person or team lead who is accountable. - **Skipping "where we got lucky"**. This section surfaces near-misses that deserve preventive action even though they didn't cause damage this time. ## Source & license This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code. - **Author:** [The-AI-Directory-Company](https://github.com/The-AI-Directory-Company) - **Source:** [The-AI-Directory-Company/agents-and-skills](https://github.com/The-AI-Directory-Company/agents-and-skills) - **License:** MIT - **Homepage:** https://ai-directory.company Install and usage instructions live in the source repository linked above. ## Pricing - **Free** — Free ## Security capabilities Automated source analysis of v0.1.0 — what this tool can access: - **Network access:** no - **Filesystem access:** no - **Shell / process execution:** no - **Environment & secrets:** no - **Dynamic code execution:** no *"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.* ## Versions - **0.1.0** — security scan: passed — Imported from the upstream source. ## Links - Listing page: https://agentstack.voostack.com/l/skill-the-ai-directory-company-agents-and-skills-incident-postmortem - Seller: https://agentstack.voostack.com/s/the-ai-directory-company - Browse the marketplace: https://agentstack.voostack.com/browse --- Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.