Monitoring Alerting

Monitoring & Alerting Design Review

Quick Reference

| If you need to… | Go to | |----------------------------------------|------------------------------------------| | Understand what this skill covers | §1 Scope | | Check mandatory prerequisites | §2 Mandatory Gates | | Choose review depth | §3 Depth Selection | | Handle incomplete context | §4 Degradation Modes | | Evaluate monitoring design item by item| §5 Design Checklist | | Avoid common alerting mistakes | §6 Anti-Examples | | Score the review result | §7 Scorecard | | Format review output | §8 Output Contract | | Validate rules before shipping | §5.5 Rule Validation | | Deep-dive SLI/SLO patterns | references/sli-slo-patterns.md | | Understand alert anti-patterns | references/alert-anti-patterns.md | | Routing / grouping / inhibition config | references/alertmanager-config-patterns.md |

---

§1 Scope

In scope — monitoring and alerting for production backend services:

• SLI (Service Level Indicator) definition and measurement
• SLO (Service Level Objective) target setting and error budget
• Prometheus alerting rules (PromQL, for duration, severity labels)
• Burn-rate alerting (multi-window, multi-burn-rate SLO alerts)
• Grafana dashboard design (layout, variable templating, panel types)
• Alert routing configuration (PagerDuty/OpsGenie/Slack, severity-based routing)
• Alert fatigue audit (noise ratio, actionability, deduplication)
• Label cardinality management (high-cardinality label detection)
• On-call runbook integration (alert → runbook → action mapping)

Out of scope — delegate to dedicated skills:

• Metrics/tracing instrumentation in code → go-observability-review
• Application performance profiling → go-benchmark
• Infrastructure provisioning (Prometheus/Grafana setup) → ops tooling
• Log aggregation pipeline design → separate skill

---

§2 Mandatory Gates

Execute gates sequentially. Each gate has a STOP condition.

Gate 1: Context Collection

| Item | Why it matters | If unknown | |------|----------------|------------| | Service type (API / worker / batch / data pipeline) | Determines which SLIs are relevant | Must clarify | | Current SLIs/SLOs (if any) | Building on existing or greenfield? | Assume greenfield | | Monitoring stack (Prometheus/Datadog/CloudWatch) | Query language and alert config format differ | Assume Prometheus + Grafana | | On-call routing (PagerDuty/OpsGenie/Slack/custom) | Determines alert destination config | Ask | | Traffic pattern (steady / bursty / batch / cron) | Affects alert window sizing and threshold | Must clarify | | Current alert count | Audit scope for alert fatigue | Ask; critical for review mode |

STOP: Cannot determine what the service does (no SLI candidates identifiable). Clarify before proceeding.

PROCEED: At least service type and traffic pattern known.

Gate 2: Scope Classification

| Mode | Trigger | Output | |------|---------|--------| | review | User provides existing alerts/dashboards | Findings + improvement recommendations | | design | User describes service needing monitoring | Complete SLI/SLO + alerts + dashboard spec | | audit | User wants alert fatigue / noise analysis | Actionability report + reduction plan |

STOP: Request is about code instrumentation (not alert/dashboard design). Redirect to go-observability-review.

PROCEED: Monitoring/alerting design intent confirmed.

Gate 3: Risk Classification

| Risk | Definition | Required action | |------|-----------|-----------------| | SAFE | New alert for non-critical service, dashboard addition | Standard review | | WARN | Modifying existing production alerts, changing routing | Validate no coverage gap introduced | | UNSAFE | Removing alerts, changing SLO targets, PagerDuty routing change | Impact assessment + rollback plan |

STOP: Any UNSAFE change without impact assessment.

PROCEED: Every change has risk level and mitigation.

Gate 4: Output Completeness

Before delivering output, verify all §8 Output Contract sections present. §8.9 Uncovered Risks must never be empty.

---

§3 Depth Selection

| Depth | When to use | Gates | References to load | |-------|-------------|-------|-------------------| | Lite | Single alert rule review, dashboard panel addition | 1–4 | None | | Standard | Full SLI/SLO definition, alert suite for a service | 1–4 | sli-slo-patterns.md | | Deep | Alert fatigue audit, multi-service monitoring architecture, burn-rate alerting | 1–4 | All three reference files |

Force Standard or higher when any signal appears: SLO definition, burn-rate alerting, PagerDuty/OpsGenie routing, multi-service dashboard, alert fatigue investigation, label cardinality concern.

---

§4 Degradation Modes

When context is incomplete, degrade gracefully — never guess traffic patterns.

| Available context | Mode | What you can do | What you cannot do | |-------------------|------|-----------------|-------------------| | Full (service type, SLIs, traffic, stack, routing) | Full | Complete SLI/SLO + alerts + dashboard | — | | Service type known, traffic unknown | Degraded | SLI selection + alert rules; flag threshold unknowns | Set precise thresholds, window sizes | | Only alert rules, no service context | Minimal | Static rule review (syntax, anti-patterns) | SLO alignment, routing review | | No existing monitoring (greenfield) | Planning | Propose monitoring strategy from requirements | Review existing setup |

Hard rule: Never set alert thresholds without knowing the traffic pattern. A 1% error rate alert on a 10-QPS service fires on a single error; on a 10K-QPS service it means 100 errors/sec. In Degraded mode, flag all thresholds as "requires traffic data to validate" in §8.9.

---

§5 Design Checklist

Execute every item. Mark PASS / WARN / FAIL with evidence.

5.1 SLI/SLO Foundation

1. SLIs defined for the service — every production service needs measurable SLIs. Standard SLIs by service type:

• API service: availability (success ratio), latency (p50/p95/p99), error rate
• Worker/consumer: processing rate, lag, error rate, processing latency
• Batch job: completion rate, duration, data quality
• Data pipeline: throughput, freshness, correctness

1. SLOs set with error budget — each SLI has a target (e.g., "99.9% availability over 30 days"). Error budget = 1 - SLO target (e.g., 0.1% = ~43 minutes/month of allowed downtime). SLOs must be agreed with stakeholders, not invented by engineers.

1. Burn-rate alerting for SLOs — instead of raw threshold alerts, use multi-window burn-rate alerts that fire when error budget is being consumed too fast. This dramatically reduces false positives. Load references/sli-slo-patterns.md for patterns.

5.2 Alert Rule Quality

1. Every alert is actionable — when this alert fires, is there a concrete action the on-call can take? If the answer is "look at it and hope it resolves," the alert should be a dashboard graph, not a page. Non-actionable alerts cause alert fatigue.

1. for duration prevents flapping — Prometheus for clause should be set to absorb transient spikes. Too short (e.g., for: 1m on a noisy metric) → flapping alerts. Too long (e.g., for: 30m) → delayed notification. Typical: for: 5m for warning, for: 2m for critical.

1. Severity labels match routing — alerts must have severity: critical|warning|info labels that map to routing rules. Critical → PagerDuty page. Warning → Slack channel. Info → dashboard only. Mislabeled severity causes either missed pages or unnecessary wake-ups.

1. Alert includes runbook link — every alerting rule should include an annotations.runbook_url pointing to a runbook with: what the alert means, how to diagnose, how to mitigate. Without runbooks, on-call responders waste time Googling their own alerts.

5.3 Dashboard Design

1. Dashboard follows USE/RED method — organize dashboards by signal type:

• USE (infrastructure): Utilization, Saturation, Errors (CPU, memory, disk, network)
• RED (services): Rate, Errors, Duration (request rate, error rate, latency)
• Top row: golden signals overview. Detail rows: drill-down by endpoint/consumer/partition.

1. Dashboard uses variables for templating — Grafana variables ($service, $namespace, $instance) allow one dashboard to serve multiple instances. Avoid hardcoded label values in queries.

1. No high-cardinality labels in dashboard queries — labels like user_id, request_id, trace_id in PromQL queries explode time series count and crash Prometheus. Use bounded labels: method, status_code, endpoint (allowlisted).

5.4 Operations & Routing

1. Alert routing matches severity — critical alerts page on-call (PagerDuty/OpsGenie with escalation). Warning alerts go to team Slack channel. Info alerts are dashboard-only. No unrouted alerts.

1. Deduplication and grouping configured — Alertmanager group_by prevents firing 100 instances of the same alert. group_wait and group_interval control batch notification timing. Without grouping, a single incident generates N alerts for N instances.

1. Inhibition rules prevent alert cascade — if the database is down, suppress all "elevated error rate" alerts from services that depend on it. Without inhibition, one root cause generates dozens of symptomatic alerts.

1. Alert fatigue metrics tracked — measure: total alerts/week, alerts-per-on-call-shift, % of alerts that required action, MTTA (mean time to acknowledge). Target: 80% actionability rate.

5.5 Rule Validation (Eat Your Own Dog Food)

A skill that preaches "no false positives, no silent gaps" must validate its rules mechanically, not by eyeball:

1. Rules pass promtool check rules — syntax + PromQL validation before any rule ships:

``bash promtool check rules alerts.yml ``

1. SLO-critical alerts have promtool test rules unit tests — Prometheus natively supports asserting "given this input series, this alert fires (or stays silent)". This is the executable form of the sensitivity/specificity tradeoff:

```yaml # alertstest.yml — run with: promtool test rules alertstest.yml rule_files:

• alerts.yml

evaluation_interval: 1m tests:

• interval: 1m

input_series:

• series: 'httperrorstotal{job="api"}'

values: '0+10x10' # 10 errors/min

• series: 'httprequeststotal{job="api"}'

values: '0+100x10' # 100 req/min → 10% error rate alertruletest:

• eval_time: 10m

alertname: HighErrorRate exp_alerts:

• exp_labels: {severity: critical, job: api}

`` Write at least two cases per SLO-critical alert: one where it MUST fire, one where it MUST stay silent (transient spike absorbed by for`).

1. Routing config passes amtool check-config — Alertmanager routing/inhibition changes are validated before deploy:

``bash amtool check-config alertmanager.yml ``

If promtool/amtool are unavailable in the environment, state Not run — unavailable in §8.4 and list the exact commands for the user; never claim rules are validated without running them.

---

§6 Anti-Examples

AE-1: Alert on absolute count instead of rate

# WRONG: fires when 10 errors exist (even over 24 hours = normal)
- alert: HighErrorCount
  expr: http_errors_total > 10
# RIGHT: alert on error rate relative to traffic
- alert: HighErrorRate
  expr: rate(http_errors_total[5m]) / rate(http_requests_total[5m]) > 0.01
  for: 5m

AE-2: No for duration — flapping on transient spike

# WRONG: fires immediately on any spike, resolves in seconds, pages at 3AM
- alert: HighLatency
  expr: histogram_quantile(0.99, rate(http_duration_seconds_bucket[5m])) > 1
# RIGHT: require sustained condition
- alert: HighLatency
  expr: histogram_quantile(0.99, rate(http_duration_seconds_bucket[5m])) > 1
  for: 5m

AE-3: Alert without runbook — on-call doesn't know what to do

# WRONG: no annotations, no runbook
- alert: DatabaseConnectionPoolExhausted
  expr: db_pool_active >= db_pool_max
# RIGHT: include runbook and summary
- alert: DatabaseConnectionPoolExhausted
  expr: db_pool_active >= db_pool_max
  for: 2m
  annotations:
    summary: "DB connection pool exhausted on {{ $labels.instance }}"
    runbook_url: "https://wiki.example.com/runbooks/db-pool-exhausted"

AE-4: user_id in PromQL label — cardinality explosion

# WRONG: unique label per user → millions of time series
- record: user_request_duration
  expr: histogram_quantile(0.99, rate(http_duration_seconds_bucket{user_id!=""}[5m]))
# RIGHT: use bounded labels only (method, status_code, endpoint)

AE-5: Critical alert routed to Slack only — no page

# WRONG: critical alert goes to Slack where it drowns in messages
route:
  receiver: slack-team
  routes:
    - match: {severity: critical}
      receiver: slack-team  # should be pagerduty!
# RIGHT: critical → PagerDuty, warning → Slack

AE-6: Monitoring gap reported as "system is stable"

WRONG: "No alerts fired this month, so the system is healthy"
RIGHT: "No alerts fired — verify alert coverage: are SLIs measured? Are thresholds correct? Absence of alerts ≠ absence of problems"

Extended anti-examples (AE-7 through AE-13) in references/alert-anti-patterns.md.

---

§7 Monitoring Scorecard

Critical — any FAIL means overall FAIL

• [ ] SLIs defined and measured for the service (availability, latency, error rate minimum)
• [ ] Every alert is actionable (clear action when it fires, not just "investigate")
• [ ] Alert severity labels match routing (critical → page, warning → Slack, info → dashboard)

Standard — 4 of 5 must pass

• [ ] SLOs set with error budget and stakeholder agreement
• [ ] for duration set on all alerts to prevent flapping
• [ ] Runbook link included in alert annotations
• [ ] Dashboard follows USE/RED method with variable templating
• [ ] No high-cardinality labels in alert rules or dashboard queries

Hygiene — 3 of 4 must pass

• [ ] Alert grouping and deduplication configured (Alertmanager group_by)
• [ ] Inhibition rules prevent alert cascade from single root cause
• [ ] Alert fatigue metrics tracked (alerts/week, actionability rate)
• [ ] Burn-rate alerting for SLO-critical services

Verdict: X/12; Critical: Y/3; Standard: Z/5; Hygiene: W/4. PASS requires: Critical 3/3 AND Standard ≥4/5 AND Hygiene ≥3/4.

---

§8 Output Contract

Every monitoring review MUST produce these sections. Write "N/A — [reason]" if inapplicable.

### 8.1 Context Gate
| Item | Value | Source |

### 8.2 Depth & Mode
[Lite/Standard/Deep] × [review/design/audit] — [rationale]

### 8.3 SLI/SLO Definition (Standard/Deep)
| SLI | Measurement | SLO Target | Error Budget |

### 8.4 Alert Rules
- Per alert: name, expr, for, severity, summary, runbook_url
- Validation evidence: `promtool check rules` / `promtool test rules` output,
  or `Not run —  unavailable` + exact commands for the user

### 8.5 Dashboard Spec (Standard/Deep)
- Panel layout, queries, variables, drill-down structure

### 8.6 Routing Configuration
- Severity → destination mapping
- Grouping, deduplication, inhibition rules

### 8.7 Alert Fatigue Assessment (audit mode)
- Total alerts/week, actionability %, recommendations

### 8.8 Runbook Mapping
| Alert | Runbo

…

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [johnqtcg](https://github.com/johnqtcg)
- **Source:** [johnqtcg/awesome-skills](https://github.com/johnqtcg/awesome-skills)
- **License:** MIT
- **Homepage:** https://johnqtcg.github.io/awesome-skills/

Install and usage instructions live in the source repository linked above.