Observability Plan

observability-plan

Designs the telemetry that makes a service debuggable at 3am and detectable before users complain — SLOs measured from the user's experience, alerts that fire only when a human must act, and the three pillars each used for what they're good at. Expands the one-line observability requirement from nfr-spec into a full operating plan; upstream of incident-comms and postmortem-generator (you can't communicate or analyze an incident you couldn't detect). Distinct from success-metrics (business outcomes) — this is operational health.

How to respond

1. Start from user journeys, not from the resource dashboard. The first artifact is the list of critical journeys ("log in", "place order", "load feed") — because an SLI measures what the user experiences, and CPU graphs measure what's easy. Ask at most 2 questions, spent on the most critical journey and what "down" means to a user there. Resource metrics (the golden signals) still get covered — but as saturation/cause signals, never as the SLO.

1. Define SLIs as good-events / valid-events from the user's side (catalog in [reference.md](reference.md)) — availability (success rate), latency (a threshold, measured at a percentile: "% of requests 80%" thresholds, for SLO-backed surfaces.

• The three pillars are split by purpose, with high-cardinality identifiers kept off metrics. Logs are structured and budgeted.
• A cardinality/cost budget exists. Observability without a cost ceiling becomes the surprise invoice.
• Dashboards are defined by the question they answer, not by quantity.

When to use this skill

• ✅ Standing up monitoring/SLOs for a new or existing service
• ✅ "What should we measure / alert on?" / designing an alerting strategy
• ✅ Fixing alert fatigue — too many pages, most non-actionable
• ✅ Turning nfr-spec's observability requirement into an implementable plan

When NOT to use this skill

• ❌ Business/product success metrics — that's success-metrics
• ❌ Customer/stakeholder comms during an outage — that's incident-comms
• ❌ The post-incident analysis — that's postmortem-generator
• ❌ The one-line observability requirement in a spec — that's nfr-spec; this is the full design beneath it

Anti-patterns to avoid

• ❌ Resource metrics as SLOs. Alerting on CPU/memory/disk as if they were user pain — they auto-recover, they spike harmlessly, and they page you at 3am for nothing while the actual user-facing error goes unnoticed.
• ❌ Averages for latency. "Average response 120ms" with a p99 of 4s means a lot of users are suffering invisibly. Measure the percentile and the threshold.
• ❌ Cause-based paging. A page for every component hiccup. Page on the symptom (users hurt / budget burning fast); let causes inform diagnosis, not wake people.
• ❌ The alert with no runbook. Firing a page that says "high error rate" with no "here's what to check" — that's a puzzle handed to someone half-asleep.
• ❌ Unbounded metric cardinality. user_id or request_id as a metric label — millions of time series, a five-figure bill, and a dashboard that times out. Those belong in logs/traces.
• ❌ String-soup logs. Unstructured printf logs you can't query or aggregate, retained forever at full volume. Structure them, sample the hot path, budget the retention.
• ❌ The 100-graph dashboard. Built to look thorough, used by no one, because no panel answers a question anyone asks under pressure.
• ❌ 99.99% by reflex. Copying a target off a blog instead of from the down-for-an-hour cost calibration; every nine multiplies the engineering and on-call burden.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: sananthanarayan
• Source: sananthanarayan/skilldrop
• License: MIT

Install and usage instructions live in the source repository linked above.