# Implementing Observability

> Monitoring, logging, and tracing implementation using OpenTelemetry as the unified standard. Use when building production systems requiring visibility into performance, errors, and behavior. Covers OpenTelemetry (metrics, logs, traces), Prometheus, Grafana, Loki, Jaeger, Tempo, structured logging (structlog, tracing, slog, pino), and alerting.

- **Type:** Skill
- **Install:** `agentstack add skill-ancoleman-ai-design-components-implementing-observability`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [ancoleman](https://agentstack.voostack.com/s/ancoleman)
- **Installs:** 0
- **Category:** [AI & ML](https://agentstack.voostack.com/c/ai-and-ml)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [ancoleman](https://github.com/ancoleman)
- **Source:** https://github.com/ancoleman/ai-design-components/tree/main/skills/implementing-observability

## Install

```sh
agentstack add skill-ancoleman-ai-design-components-implementing-observability
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Production Observability with OpenTelemetry

## Purpose

Implement production-grade observability using OpenTelemetry as the 2025 industry standard. Covers the three pillars (metrics, logs, traces), LGTM stack deployment, and critical log-trace correlation patterns.

## When to Use

Use when:
- Building production systems requiring visibility into performance and errors
- Debugging distributed systems with multiple services
- Setting up monitoring, logging, or tracing infrastructure
- Implementing structured logging with trace correlation
- Configuring alerting rules for production systems

Skip if:
- Building proof-of-concept without production deployment
- System has  Result {
    // trace_id/span_id automatically included
    info!(user_id = user_id, "processing request");
    Ok(result)
}
```

**See**: `references/trace-context.md` for Go and TypeScript patterns.

### Query in Grafana

```logql
{job="api-service"} |= "trace_id=4bf92f3577b34da6a3ce929d0e0e4736"
```

## Quick Setup Guide

### 1. Choose Your Stack

**Decision Tree**:
- **Greenfield**: OpenTelemetry SDK + LGTM Stack (self-hosted) or Grafana Cloud (managed)
- **Existing Prometheus**: Add Loki (logs) + Tempo (traces)
- **Kubernetes**: LGTM via Helm, Alloy DaemonSet
- **Zero-ops**: Managed SaaS (Grafana Cloud, Datadog, New Relic)

### 2. Install OpenTelemetry SDK

**Bootstrap Script**:
```bash
python scripts/setup_otel.py --language python --framework fastapi
```

**Manual (Python)**:
```bash
pip install opentelemetry-api opentelemetry-sdk \
    opentelemetry-instrumentation-fastapi \
    opentelemetry-exporter-otlp
```

**See**: `references/opentelemetry-setup.md` for Rust, Go, TypeScript installation.

### 3. Deploy LGTM Stack

**Docker Compose** (development):
```bash
cd examples/lgtm-docker-compose
docker-compose up -d
# Grafana: http://localhost:3000 (admin/admin)
# OTLP: localhost:4317 (gRPC), localhost:4318 (HTTP)
```

**See**: `references/lgtm-stack.md` for production Kubernetes deployment.

### 4. Configure Structured Logging

**See**: `references/structured-logging.md` for complete setup (Python, Rust, Go, TypeScript).

### 5. Set Up Alerting

**See**: `references/alerting-rules.md` for Prometheus and Loki alert patterns.

## Auto-Instrumentation

OpenTelemetry auto-instruments popular frameworks:

```python
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor

app = FastAPI()
FastAPIInstrumentor.instrument_app(app)  # Auto-trace all HTTP requests
```

**Supported**: FastAPI, Flask, Django, Express, Gin, Echo, Nest.js

**See**: `references/opentelemetry-setup.md` for framework-specific setup.

## Common Patterns

### Custom Spans

```python
from opentelemetry import trace

tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("fetch_user_details") as span:
    span.set_attribute("user_id", user_id)
    user = await db.fetch_user(user_id)
    span.set_attribute("user_found", user is not None)
```

### Error Tracking

```python
from opentelemetry.trace import Status, StatusCode

with tracer.start_as_current_span("process_payment") as span:
    try:
        result = process_payment(amount, card_token)
        span.set_status(Status(StatusCode.OK))
    except PaymentError as e:
        span.set_status(Status(StatusCode.ERROR, str(e)))
        span.record_exception(e)
        raise
```

**See**: `references/trace-context.md` for background job tracing and context propagation.

## Validation and Testing

```bash
# Test log-trace correlation
# 1. Make request to your app
# 2. Copy trace_id from logs
# 3. Query in Grafana: {job="myapp"} |= "trace_id="

# Validate metrics
python scripts/validate_metrics.py
```

## Integration with Other Skills

- **Dashboards**: Embed Grafana panels, query Prometheus metrics
- **Feedback**: Alert routing (Slack, PagerDuty), notification UI
- **Data-Viz**: Time-series charts, trace waterfall, latency heatmaps

**See**: `examples/fastapi-otel/` for complete integration.

## Progressive Disclosure

**Setup Guides**:
- `references/opentelemetry-setup.md` - SDK installation (Python, Rust, Go, TypeScript)
- `references/structured-logging.md` - structlog, tracing, slog, pino configuration
- `references/lgtm-stack.md` - LGTM deployment (Docker, Kubernetes)
- `references/trace-context.md` - Log-trace correlation patterns
- `references/alerting-rules.md` - Prometheus and Loki alert templates

**Examples**:
- `examples/fastapi-otel/` - FastAPI + OpenTelemetry + LGTM
- `examples/axum-tracing/` - Rust Axum + tracing + LGTM
- `examples/lgtm-docker-compose/` - Production-ready LGTM stack

**Scripts**:
- `scripts/setup_otel.py` - Bootstrap OpenTelemetry SDK
- `scripts/generate_dashboards.py` - Generate Grafana dashboards
- `scripts/validate_metrics.py` - Validate metric naming

## Key Principles

1. **OpenTelemetry is THE standard** - Use OTel SDK, not vendor-specific SDKs
2. **Auto-instrumentation first** - Prefer auto over manual spans
3. **Always correlate logs and traces** - Inject trace_id/span_id into every log
4. **Use structured logging** - JSON format, consistent field names
5. **LGTM stack for self-hosting** - Production-ready open-source stack

## Common Pitfalls

**Don't**:
- Use vendor-specific SDKs (use OpenTelemetry)
- Log without trace_id/span_id context
- Manually instrument what auto-instrumentation covers
- Mix logging libraries (pick one: structlog, tracing, slog, pino)

**Do**:
- Start with auto-instrumentation
- Add manual spans only for business-critical operations
- Use semantic conventions for span attributes
- Export to OTLP (gRPC preferred over HTTP)
- Test locally with LGTM docker-compose before production

## Success Metrics

1. 100% of logs include trace_id when in request context
2. Mean time to resolution (MTTR) decreases by >50%
3. Developers use Grafana as first debugging tool
4. 80%+ of telemetry from auto-instrumentation
5. Alert noise < 5% false positives

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [ancoleman](https://github.com/ancoleman)
- **Source:** [ancoleman/ai-design-components](https://github.com/ancoleman/ai-design-components)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-ancoleman-ai-design-components-implementing-observability
- Seller: https://agentstack.voostack.com/s/ancoleman
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.