# Observability Planner

> >

- **Type:** Skill
- **Install:** `agentstack add skill-patonkikh-apes-observability-planner`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [patonkikh](https://agentstack.voostack.com/s/patonkikh)
- **Installs:** 0
- **Category:** [AI & ML](https://agentstack.voostack.com/c/ai-and-ml)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [patonkikh](https://github.com/patonkikh)
- **Source:** https://github.com/patonkikh/APES/tree/main/skills/architecture/observability-planner

## Install

```sh
agentstack add skill-patonkikh-apes-observability-planner
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Observability Planner

# Purpose

Design observability for AI-powered systems: what to trace, measure, log, alert on, and display — aligned to SLOs and incident response.

**Input:** System architecture, SLO targets, critical user journeys, stack constraints (cloud, APM vendor)  
**Output:** Observability Plan with signal catalog, dashboard specs, alert rules, and runbook hooks  
**Examples:** See [examples.md](examples.md) for worked input/output.

---

# Workflow

## Step 1: Define SLIs and SLOs

| Journey | SLI | SLO example |
|---------|-----|-------------|
| Chat response | Availability + latency | 99.5% 1% |
| Prompt version | `prompt_version` attribute | N/A (dimension) |
| Cost | `token_count` × price | daily budget |
| Quality proxy | User rating event | rating drop >10% |

Correlate with `analytics-instrumentation-planner` product events.

## Step 4: Design dashboards

| Dashboard | Audience | Panels |
|-----------|----------|--------|
| Service health | On-call | SLO, error rate, latency heatmap |
| AI pipeline | AI engineers | Token usage, model mix, RAG stats |
| Business overlay | Product | DAU + error rate correlation |

Each panel links to trace exemplar for drill-down.

## Step 5: Define alerts and runbooks

| Alert | Condition | Severity | Runbook |
|-------|-----------|----------|---------|
| High 5xx | >1% 5min | P1 | Scale + rollback |
| LLM timeout | p95 > SLO 10min | P2 | Model fallback |
| Ingest lag | >1h | P2 | Queue drain playbook |
| Cost spike | +50% daily | P3 | Rate limit review |

Alerts must be actionable; tie to owner and escalation path.

## Step 6: Logging and privacy

- Structured JSON logs with `trace_id`, `user_id` (hashed), `request_id`
- Never log raw prompts/responses in prod; use opt-in debug tier
- Retention: hot 7d, cold 30d, compliance per policy

## Step 7: Validate

Run Validation checklist.

---

# Decision Rules

| Condition | Action |
|-----------|--------|
| No SLO defined | Stop; propose SLIs from critical journeys |
| Multi-tenant | Tag all signals with `tenant_id` |
| Third-party LLM only | Trace at client; log provider request IDs |
| Regulated data | No full payload logging; spans only |
| Serverless | Prefer metrics + traces; minimize log volume cost |

---

# Validation

- [ ] SLIs/SLOs per critical journey
- [ ] Trace spans for LLM, RAG, and tools defined
- [ ] Metrics catalog with names and labels
- [ ] ≥2 dashboards specified with audience
- [ ] Alerts with thresholds and runbook links
- [ ] Privacy rules for prompts and PII in logs
- [ ] Error budget policy documented
- [ ] Correlation ID propagated end-to-end

---

# Anti-patterns

- **Logs only** — cannot debug distributed LLM latency without traces.
- **Alert fatigue** — 50 thresholds nobody acts on.
- **Dashboard vanity** — charts without SLO line.
- **Logging prompts in prod** — security and compliance risk.
- **Missing business metrics** — ops green while users churn.

---

# Best Practices

- Adopt OpenTelemetry early; one instrumentation for metrics + traces.
- Store `prompt_version` and `model_id` on every LLM span.
- Practice game days: inject failure, verify alerts fire.
- Pair with `ai-evaluation-builder` for quality regressions.
- Review observability coverage in architecture review checklist.

---

# Output Structure

```markdown
# Observability Plan: [System Name]

## SLOs
| Journey | SLI | Target | Error budget |
|---------|-----|--------|--------------|

## Signal Catalog
### Traces
[Span list + attributes]

### Metrics
[Name, labels, collection]

### Logs
[Fields, retention]

## Dashboards
[Panel descriptions]

## Alerts
| Name | Query | Threshold | Runbook |

## Privacy
[Logging restrictions]

## Rollout
[Instrumentation phases]
```

---

# Next Skills

| Outcome | Recommended Skill |
|---------|-------------------|
| Product-side metrics | `product/analytics-instrumentation-planner` |
| AI eval regression gates | `ai/ai-evaluation-builder` |
| Latency optimization | `ai/ai-latency-optimizer` |
| Cost dashboards | `ai/ai-cost-optimizer` |
| Architecture review | `architecture/architecture-review` |

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [patonkikh](https://github.com/patonkikh)
- **Source:** [patonkikh/APES](https://github.com/patonkikh/APES)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-patonkikh-apes-observability-planner
- Seller: https://agentstack.voostack.com/s/patonkikh
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.