Incident

Incident Response

During an Incident

1. Triage (First 5 minutes)

• What's broken? User-facing impact. Who's affected and how many.
• When did it start? Check monitoring, deployment timeline, recent changes.
• Is it getting worse? Stable, degrading, or cascading.
• Severity?

| Severity | Impact | Response | |----------|--------|----------| | SEV1 | Service down, data loss risk | All hands, immediate | | SEV2 | Major feature broken, many users affected | Drop current work | | SEV3 | Minor feature broken, workaround exists | Next working hours | | SEV4 | Cosmetic, edge case | Normal priority |

2. Mitigate (Stop the bleeding)

Priority: restore service, not find root cause. Options:

• Rollback the last deployment
• Feature flag disable the broken feature
• Scale up if it's a capacity issue
• Redirect traffic away from broken component
• Restart if it's a stuck process or resource leak

Document what you do and when. Timestamps matter.

3. Communicate

• Notify affected users/stakeholders with honest status
• Set expectations for next update ("we'll update in 30 minutes")
• Don't speculate on cause until you know

4. Investigate (After service is restored)

• Gather logs, metrics, traces from the incident window
• Correlate with deployment timeline and config changes
• Identify the triggering event and the root cause (they're often different)

Post-Mortem Template

Write within 48 hours while memory is fresh. Blameless - focus on systems, not people.

# Incident: [Title]
Date: [YYYY-MM-DD]
Duration: [start time → end time]
Severity: [SEV1-4]
Author: [who's writing this]

## Summary
[2-3 sentences: what happened, who was affected, how it was resolved]

## Timeline
- HH:MM - [event]
- HH:MM - [event]
- HH:MM - [detection / alert fired]
- HH:MM - [response began]
- HH:MM - [mitigation applied]
- HH:MM - [service restored]

## Root Cause
[What actually broke and why. Be specific.]

## Contributing Factors
- [Factor 1: why it was possible for this to happen]
- [Factor 2: why it wasn't caught earlier]
- [Factor 3: why recovery took as long as it did]

## What Went Well
- [Things that helped during response]

## What Went Poorly
- [Things that made response harder]

## Action Items
| Action | Owner | Priority | Deadline |
|--------|-------|----------|----------|
| [Prevent recurrence] | ? | P1 | ? |
| [Improve detection] | ? | P2 | ? |
| [Improve response] | ? | P3 | ? |

## Lessons Learned
[What should the team internalize from this?]

Prevention Patterns

| Pattern | Prevents | |---------|----------| | Canary deploys | Bad code reaching all users | | Health checks | Serving traffic from broken instances | | Circuit breakers | Cascading failures | | Rate limiting | Resource exhaustion | | Automated rollback | Prolonged outages from bad deploys | | Chaos testing | Surprises in production | | Runbooks | Slow response from unfamiliarity |

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: pvnarp
• Source: pvnarp/agent-skills
• License: MIT

Install and usage instructions live in the source repository linked above.