Slo Implementation

Skill: SLO Implementation

> Expertise: Prometheus recording rules for SLOs, multi-window burn rate alerts, Sloth code generation, error budget Grafana panels.

When to load

When implementing SLOs for a service in Prometheus, setting up burn rate alerts, or creating error budget dashboards.

Full SLO Stack (single service)

Step 1: Define the SLI Recording Rules

# prometheus-rules/slo-checkout-service.yaml
groups:
  - name: slo:checkout-service:recording
    interval: 30s
    rules:
      # Good requests: 2xx, latency  14.4× baseline"
          description: "1h availability: {{ $value | humanizePercentage }}. Budget burning rapidly."
          runbook_url: "https://runbooks.internal/checkout-slo-fast-burn"

      # ── Slow burn (6h + 30m windows, 6× rate) ────────────────────
      # Consumes 5% of 28d budget in 6h → ticket, fix in business hours
      - alert: CheckoutSLOSlowBurn
        expr: |
          (slo:http_availability:ratio_rate6h{service="checkout-service"}  6× baseline"
          runbook_url: "https://runbooks.internal/checkout-slo-slow-burn"

      # ── Budget exhaustion warning ─────────────────────────────────
      - alert: CheckoutSLOBudgetLow
        expr: |
          slo:http_availability:ratio_rate28d{service="checkout-service"}
          < (1 - 0.005 * 0.75)   # < 25% budget remaining
        for: 1h
        labels:
          severity: warning
          service: checkout-service
        annotations:
          summary: "Checkout error budget < 25% remaining for this month"
          runbook_url: "https://runbooks.internal/checkout-error-budget"

Step 3: Sloth (generate from YAML spec)

# slo/checkout-service.yaml
version: "prometheus/v1"
service: checkout-service
labels: { team: backend, tier: "1" }
slos:
  - name: requests-availability
    objective: 99.5
    description: "99.5% of checkout requests succeed"
    sli:
      events:
        error_query: |
          sum(rate(http_requests_total{
            service="checkout-service",
            status=~"5.."}[{{.window}}]))
        total_query: |
          sum(rate(http_requests_total{
            service="checkout-service"}[{{.window}}]))
    alerting:
      name: CheckoutServiceAvailability
      page_alert:
        labels: { severity: critical }
        annotations:
          runbook_url: https://runbooks.internal/checkout-availability
      ticket_alert:
        labels: { severity: warning }

# Generate Prometheus rules + alerts from Sloth spec
sloth generate -i slo/checkout-service.yaml -o rules/slo-checkout-generated.yaml
# Produces: recording rules for all windows + multi-window burn rate alerts

Step 4: Error Budget Dashboard (Grafana)

-- Current error budget remaining (percent of 28d budget)
(
  sum_over_time(slo:http_availability:ratio_rate5m{service="checkout-service"}[28d])
  / (28 * 24 * 12)
  - (1 - 0.005)
)
/ 0.005 * 100

-- Hours of budget remaining at current burn rate
(
  (slo:http_availability:ratio_rate28d{service="checkout-service"} - (1 - 0.005))
  / 0.005
) * 28 * 24

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: sawrus
• Source: sawrus/agent-guides
• License: MIT
• Homepage: https://sawrus.github.io/agent-guides

Install and usage instructions live in the source repository linked above.