Incident Management

Incident Management

Implement effective incident management processes including severity definitions, escalation matrices, war room procedures, and blameless post-mortem templates.

When to Use

• Establishing incident management processes for production systems
• Defining severity levels and escalation procedures
• Running war rooms and coordinating incident response
• Conducting blameless post-incident reviews
• Building on-call schedules and notification workflows
• Meeting compliance requirements for incident response (SOC 2, HIPAA, PCI DSS)

Severity Levels

severity_definitions:
  SEV1_critical:
    impact: "Complete service outage or data breach affecting all/most customers"
    examples:
      - Production site completely down
      - Data breach confirmed or suspected
      - Complete loss of a critical business function
      - Security incident with active exploitation
    response_time: "Immediate (within 5 minutes)"
    update_frequency: "Every 15-30 minutes"
    who_is_paged: "On-call engineer, engineering manager, incident commander, executive on-call"
    communication: "Status page update, customer email, executive notification"
    resolution_target: "5x latency)
      - Data processing pipeline completely stalled
      - Partial outage affecting a region or segment
    response_time: "Within 15 minutes"
    update_frequency: "Every 30-60 minutes"
    who_is_paged: "On-call engineer, engineering manager"
    communication: "Status page update if customer-facing"
    resolution_target: " 90% completed on time"
    trending: "Monitors follow-through on improvements"

  recurring_incidents:
    name: "Percentage of incidents with same root cause as previous incident"
    target: "< 10%"
    trending: "Monitors effectiveness of preventive actions"

Incident Management Checklist

incident_management_checklist:
  process_setup:
    - [ ] Severity levels defined with clear criteria
    - [ ] Escalation matrix documented
    - [ ] On-call schedule established and staffed
    - [ ] War room procedures documented
    - [ ] Post-mortem template created
    - [ ] Communication templates prepared (status page, email)
    - [ ] Incident management tool configured (PagerDuty, OpsGenie)

  per_incident:
    - [ ] Incident declared with severity level
    - [ ] War room established (SEV1/2)
    - [ ] Roles assigned (IC, comms, technical lead, scribe)
    - [ ] Timeline maintained in real-time
    - [ ] Status page updated (customer-facing impact)
    - [ ] Stakeholders notified per communication plan
    - [ ] Resolution verified with monitoring
    - [ ] Post-mortem scheduled
    - [ ] Post-mortem conducted and published
    - [ ] Action items tracked to completion

  compliance:
    - [ ] All SEV1/2 incidents have post-mortems
    - [ ] Incident log maintained for audit evidence
    - [ ] Metrics reported monthly
    - [ ] On-call health monitored (pages per person)
    - [ ] Annual incident response training conducted
    - [ ] Annual incident response plan test completed

Best Practices

• Define severity levels with concrete examples so there is no ambiguity during an active incident
• Implement time-based escalation: if the on-call does not acknowledge, automatically escalate
• Focus on mitigation first, root cause second: restore service before investigating why it failed
• Run blameless post-mortems: the goal is to improve systems, not to assign fault to individuals
• Track post-mortem action items to completion: an unfinished action item means the same incident can recur
• Monitor incident metrics (MTTR, MTTD, frequency) as leading indicators of system reliability
• Protect on-call health: track page volume per person and redistribute if someone is overburdened
• Separate the incident commander role from the technical lead role in SEV1/2 incidents
• Practice incident response regularly with game days or chaos engineering exercises
• Archive incident records and post-mortems for compliance evidence and organizational learning

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: BagelHole
• Source: BagelHole/DevOps-Security-Agent-Skills
• License: MIT

Install and usage instructions live in the source repository linked above.