# Instrument Observability

> Instrument a change with structured logging, RED metrics, trace spans, and symptom-based alerts - use while building the change, not after an incident proves it was unobservable

- **Type:** Skill
- **Install:** `agentstack add skill-etr-groundwork-instrument-observability`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [etr](https://agentstack.voostack.com/s/etr)
- **Installs:** 0
- **Category:** [Data & Analytics](https://agentstack.voostack.com/c/data-and-analytics)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [etr](https://github.com/etr)
- **Source:** https://github.com/etr/groundwork/tree/main/skills/instrument-observability

## Install

```sh
agentstack add skill-etr-groundwork-instrument-observability
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Instrument Observability

## Overview

Observability is added with the code that needs it, not bolted on after an outage. **Shift Left:** the cheapest time to make a change observable is while you still hold its context — what can fail, what "normal" looks like, which boundary the latency lives behind.

**Core principle:** A change you cannot observe in production is a change you cannot operate. Logs, metrics, traces, and alerts are part of "done," not a follow-up ticket.

## When to Use

- Adding or changing a service boundary, request handler, job, or external call
- Code paths that can fail partially, retry, time out, or degrade
- Any change whose health you would want to confirm during a [[staged-rollout]]

Skip only for changes with no runtime behavior (docs, pure refactors with identical I/O, config-only edits).

## Process

Apply each layer to the change.

1. **Structured logging.** Emit machine-parseable events (key-value / JSON), not interpolated prose. Attach the correlation/trace id and the dimensions you'd filter on (tenant, route, outcome). Log decisions and failures, not control flow.
2. **RED metrics.** For every request-serving surface, instrument the three:
   - **Rate** — requests per second handled
   - **Errors** — failed requests per second (and the error class)
   - **Duration** — latency distribution (histogram, so you get p50/p95/p99 — never a single mean)
3. **Trace spans.** Wrap each external boundary (DB query, RPC, queue, third-party API) in a span that propagates context. Spans turn "the request was slow" into "the request was slow *here*."
4. **Symptom-based alerts.** Alert on user-visible symptoms (error rate breached, latency SLO burning), **not** on causes (CPU high, pod restarted). Causes generate noise; symptoms generate pages worth waking for. Each alert names the symptom and points at the dashboard/runbook.

## Rationalizations

| Excuse | Reality |
|--------|---------|
| "I'll add metrics once it's in prod" | The first incident is the worst time to discover you're blind. Shift Left. |
| "The mean latency is fine" | A mean hides the p99 tail where users actually hurt. Use a histogram. |
| "There are already logs" | Unstructured logs you can't query are not observability. Structure them. |
| "I'll alert on CPU and disk" | Cause-based alerts page you for non-problems and miss real ones. Alert on symptoms. |
| "Tracing is a separate project" | One span around each boundary is minutes of work and the only thing that localizes latency. |

## Red Flags

- Latency reported as a single average instead of a distribution
- Alerts wired to CPU, memory, or restart counts rather than user-visible symptoms
- `log.info("processing " + thing)` style string interpolation instead of structured fields
- An external call (DB, API, queue) with no span around it
- "We'll add observability after launch" in the plan

## Verification

Cite concrete evidence — names and locations, not intentions:

- [ ] Structured log events added at decision/failure points, carrying the correlation id (name the events)
- [ ] Rate, Error, and Duration metrics emitted for each request surface; Duration is a histogram (name the metrics)
- [ ] A trace span wraps every external boundary the change touches (name the spans)
- [ ] At least one symptom-based alert defined against an SLO, pointing at a dashboard/runbook (name the alert)

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [etr](https://github.com/etr)
- **Source:** [etr/groundwork](https://github.com/etr/groundwork)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-etr-groundwork-instrument-observability
- Seller: https://agentstack.voostack.com/s/etr
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.