Monitoring

Monitoring & Alerting

Monitor for user impact, not system internals. A server at 95% CPU is fine if users are happy. A server at 20% CPU is broken if requests are timing out.

The Four Golden Signals

Monitor these for every service:

| Signal | What It Measures | Example Metric | |--------|-----------------|----------------| | Latency | How long requests take | p50, p95, p99 response time | | Traffic | How much demand | Requests per second, concurrent users | | Errors | How often things fail | Error rate (%), 5xx count | | Saturation | How full the system is | CPU, memory, disk, queue depth, connection pool |

If you can only monitor four things, monitor these.

SLOs, SLIs, SLAs

SLI (Service Level Indicator)

A metric that measures a user-facing behavior.

SLI: Percentage of HTTP requests completing in  5 minutes

### When NOT to Alert
- CPU/memory/disk at a threshold (alert on user impact, not system state)
- A single transient error (wait for pattern)
- Something that can wait until business hours (use a ticket instead)
- A dashbaord already shows it (redundant noise)

### Alert Template

ALERT: [clear, specific name] SEVERITY: critical / warning / info CONDITION: [metric] [operator] [threshold] for [duration] IMPACT: [what users experience] RUNBOOK: [link to resolution steps] ESCALATION: [who to contact if not resolved in X minutes]

### Alert Rules
1. **Every alert must be actionable.** If the response is "wait and see," it's not an alert.
2. **Every alert needs a runbook.** No one remembers what to do at 3 AM.
3. **Review alerts monthly.** Delete alerts that fire but require no action (noise).
4. **Page on symptoms, ticket on causes.** "Requests failing" is a page. "Disk at 80%" is a ticket.

## Metric Naming Convention

...

Examples

api.http.requestduration.milliseconds api.http.requests.total api.database.queryduration.milliseconds api.database.connections.active worker.jobs.processed.total worker.jobs.failed.total worker.queue.depth.count

## Common Monitoring Stack

| Need | Tools |
|------|-------|
| Metrics | Prometheus, Datadog, CloudWatch, Grafana |
| Logs | Loki, ELK, CloudWatch Logs, Datadog Logs |
| Traces | Jaeger, Zipkin, Datadog APM, Honeycomb |
| Alerts | PagerDuty, OpsGenie, Grafana Alerting |
| Uptime | Pingdom, UptimeRobot, Checkly |
| Dashboards | Grafana, Datadog, CloudWatch |

> Reference: `reference/alerting-patterns.md` - golden signals by service type, severity levels, alert fatigue prevention, SLO/SLI templates, dashboard layouts, PromQL patterns, escalation decision tree.

## Monitoring Maturity Checklist

### Level 1: Basic
- [ ] Health check endpoint exists and is monitored
- [ ] Error rate tracked and alerted on
- [ ] Basic uptime monitoring (is it responding?)

### Level 2: Operational
- [ ] Four golden signals dashboarded per service
- [ ] Alerts have runbooks
- [ ] Deploy markers on graphs
- [ ] Log aggregation searchable

### Level 3: SLO-Driven
- [ ] SLOs defined for user-facing services
- [ ] Error budget tracking and burn rate alerts
- [ ] Distributed tracing for request flow
- [ ] Alert noise reviewed and reduced monthly

### Level 4: Proactive
- [ ] Anomaly detection on key metrics
- [ ] Capacity planning based on traffic trends
- [ ] Chaos testing validates monitoring coverage
- [ ] On-call load balanced and sustainable

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [pvnarp](https://github.com/pvnarp)
- **Source:** [pvnarp/agent-skills](https://github.com/pvnarp/agent-skills)
- **License:** MIT

Install and usage instructions live in the source repository linked above.