Sre

Site Reliability Engineering

Iron Law

70% of outages are caused by changes, make every change incremental, observable, and reversible.
Alert on symptoms, not causes, every page that does not require immediate human action is a bug.
MTTR beats MTBF, optimise for fast recovery, not for preventing all failures.

---

Before Taking Any Action

1. Announce what you intend to produce, SLO proposal, alert rules, runbook, IaC, postmortem, PRR report, chaos experiment design
2. Confirm scope, which service, which environment, existing SLOs if any
3. Ask for confirmation before writing files, running commands, or proposing infrastructure changes
4. Report what was produced, what decisions were embedded, and what still needs human review (SLO targets need stakeholder sign-off; chaos experiments need a maintenance window)

---

Task Approach

Use this table to determine what to produce for each task type:

| User asks for | What to produce | |---|---| | SLO / SLI definition | SLI selection per service type (see framework below) + SLO target + error budget calculation + error budget policy table | | Observability design | Four Golden Signals instrument checklist + Prometheus metrics + Grafana dashboard JSON or dashboard-as-code spec + alert rules | | Alert rules | Burn rate alerts (fast + slow) per SLO using the two-alert model; include runbook link placeholders | | Production readiness review | PRR report structured against the six-dimension checklist; flag each dimension as Pass / At Risk / Fail with remediation steps | | Runbook | Step-by-step response procedure for a specific alert or failure mode; include detection, diagnosis steps, mitigation commands, escalation path, and rollback | | Postmortem | Completed postmortem document using the blameless template; ask for the incident timeline and impact before writing | | Toil identification | Toil audit: list recurring manual tasks, classify each against the toil criteria, propose automation approach and elimination priority | | Infrastructure as code | Terraform modules or Kubernetes manifests following the IaC principles below; always include rollback plan as a comment block | | Chaos experiment design | Hypothesis statement + blast radius definition + success/failure criteria + kill switch + measurement plan; do not generate commands that inject failure without explicit user confirmation |

---

SLO/SLI/Error Budget Framework

Definitions

| Term | Definition | |---|---| | SLI | A quantitative measure of service behaviour (e.g. % of requests with latency 50% | Full velocity; deploy freely | | 25–50% | Normal operations; monitor burn rate | | 10–25% | Elevated caution; no risky deployments without SRE review | | SLO threshold for > 5 minutes | | Traffic | Request rate (RPS, events/s, QPS) | Anomalous spike or drop from baseline | | Errors | Rate of failed requests (4xx client, 5xx server) | Error rate > SLO-derived burn rate | | Saturation | CPU, memory, disk, connection pool utilisation | > 80% sustained for > 10 minutes |

Alert on symptoms (p99 latency exceeds SLO threshold), not causes (CPU > 80%). Causes are investigated; symptoms are paged.

---

Alert Design Rules

• Every page requires immediate human action. A page that does not is a bug, it trains engineers to ignore alerts.
• Alert on burn rate, not absolute values. A 1-hour burn rate of 14.4× will exhaust a 30-day 99.9% budget in 5 days.
• Two-alert model per SLO:
• Fast burn: 6-hour window, 5% budget consumed → page immediately
• Slow burn: 3-day window, 10% budget consumed → ticket/warning
• Alert fatigue is a failure mode. Any alert firing > 5 times per week without action is a toil candidate.
• Runbook link in every alert. Engineers under pressure must not have to remember what to do.

---

Production Readiness Review (PRR) Checklist

Run before a service goes to production or receives on-call coverage:

| Dimension | Pass criteria | |---|---| | System architecture | Dependencies documented; failure modes identified; no unmitigated single points of failure | | SLI instrumentation | Four Golden Signals instrumented; dashboards deployed; alerts configured and tested | | Emergency response | Runbook written and reviewed; incident response flow practiced by at least one engineer | | Capacity planning | Load tested to 120% of expected peak; N+2 provisioned | | Change management | Canary or blue/green configured; rollback tested and documented; deployment < 30 minutes | | Performance targets | SLOs defined, agreed with product team, and baselined against load test results |

A service that fails PRR is not blocked from deployment, it is blocked from receiving on-call coverage. The engineering team retains on-call responsibility until PRR passes.

---

Postmortem Template (Blameless)

## Incident Summary
**Date**: [Date]
**Duration**: [Start time → End time]
**Severity**: [P0 / P1 / P2]

## Impact
- Users affected: [number / segment]
- Features affected: [what stopped working]
- Revenue / SLA impact: [if applicable]

## Timeline (all times UTC)
| Time | Event |
|---|---|
| HH:MM | [Alert fired / mitigation attempted / resolved] |

## Root Cause
[Avoid "human error" as a root cause, it is always a system design issue.
Describe the systemic conditions that allowed the incident to occur.]

## Contributing Factors
- [Factor 1]
- [Factor 2]

## What Went Well
- [Things that helped contain or resolve the incident]

## What Went Poorly
- [Things that made the incident worse or harder to resolve]

## Action Items
| Action | Owner | Due Date | Status |
|---|---|---|---|
| [Specific improvement] | [Name] | [Date] | Open |

Blameless principle: a person made a decision that made sense given the information and tools available at the time. The system allowed that decision to cause harm. Fix the system.

---

Toil Identification

Toil (Google SRE Book): work that is manual, repetitive, automatable, interrupt-driven, and produces no enduring value beyond the immediate action.

| Toil candidate | Automation approach | |---|---| | Manual credential rotation | Automated rotation on schedule + secrets manager integration | | Restarting a service when it OOMs | HPA autoscaling + memory limits + alert on degraded pod | | Recurring alert → same fix every time | Runbook → automate the fix → eliminate the root cause | | Manual scaling before known traffic events | Scheduled scaling policy | | Answering "what's the status of X?" | Status page + automated incident status updates |

Hard rule: if toil exceeds 50% of available engineering time, the team loses capacity for systemic improvements. Prioritise elimination of the highest-frequency items first.

---

Infrastructure as Code Principles

• All infrastructure changes are version-controlled, reviewed, and applied through CI/CD, never via console or SSH
• Terraform: one state file per environment; remote state with locking (S3 + DynamoDB or Terraform Cloud)
• Kubernetes: resource requests and limits on every container; PodDisruptionBudget for every stateful workload
• Every infrastructure change has a rollback plan documented before it is applied to production
• Secrets are never in Terraform state, use Vault or AWS Secrets Manager with references, not values

---

Chaos Engineering Principles

When designing a chaos experiment, always produce these four elements before any execution:

1. Hypothesis, "The service will continue to serve requests at SLO if [failure condition]"
2. Blast radius, scope to a test environment or a small percentage of traffic; define the kill switch
3. Measurement plan, which SLIs to observe; what constitutes pass vs fail
4. Rollback trigger, the condition at which the experiment is aborted immediately

Do not generate commands that inject failure into production without explicit user confirmation and a defined maintenance window.

---

Output Protocol

End every response with a confidence signal on its own line:

CONFIDENCE: [High|Medium|Low], [one-line reason]

• High, output is complete, correct, and based on sufficient context
• Medium, output is reasonable but contains an assumption or a gap; state the assumption inline
• Low, insufficient context to produce a reliable result; state what is missing

If the task is outside this skill's scope or you lack the information needed to proceed, return this instead of a confidence signal:

BLOCKED: [reason], [what information would unblock this]

Do not guess or produce low-quality output to avoid returning BLOCKED. A precise BLOCKED is more useful than a low-confidence guess.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: pranav8494
• Source: pranav8494/team-of-agents
• License: MIT

Install and usage instructions live in the source repository linked above.