Implementing Observability

Production Observability with OpenTelemetry

Purpose

Implement production-grade observability using OpenTelemetry as the 2025 industry standard. Covers the three pillars (metrics, logs, traces), LGTM stack deployment, and critical log-trace correlation patterns.

When to Use

Use when:

• Building production systems requiring visibility into performance and errors
• Debugging distributed systems with multiple services
• Setting up monitoring, logging, or tracing infrastructure
• Implementing structured logging with trace correlation
• Configuring alerting rules for production systems

Skip if:

• Building proof-of-concept without production deployment
• System has Result {

// traceid/spanid automatically included info!(userid = userid, "processing request"); Ok(result) }

**See**: `references/trace-context.md` for Go and TypeScript patterns.

### Query in Grafana

```logql
{job="api-service"} |= "trace_id=4bf92f3577b34da6a3ce929d0e0e4736"

Quick Setup Guide

1. Choose Your Stack

Decision Tree:

• Greenfield: OpenTelemetry SDK + LGTM Stack (self-hosted) or Grafana Cloud (managed)
• Existing Prometheus: Add Loki (logs) + Tempo (traces)
• Kubernetes: LGTM via Helm, Alloy DaemonSet
• Zero-ops: Managed SaaS (Grafana Cloud, Datadog, New Relic)

2. Install OpenTelemetry SDK

Bootstrap Script:

python scripts/setup_otel.py --language python --framework fastapi

Manual (Python):

pip install opentelemetry-api opentelemetry-sdk \
    opentelemetry-instrumentation-fastapi \
    opentelemetry-exporter-otlp

See: references/opentelemetry-setup.md for Rust, Go, TypeScript installation.

3. Deploy LGTM Stack

Docker Compose (development):

cd examples/lgtm-docker-compose
docker-compose up -d
# Grafana: http://localhost:3000 (admin/admin)
# OTLP: localhost:4317 (gRPC), localhost:4318 (HTTP)

See: references/lgtm-stack.md for production Kubernetes deployment.

4. Configure Structured Logging

See: references/structured-logging.md for complete setup (Python, Rust, Go, TypeScript).

5. Set Up Alerting

See: references/alerting-rules.md for Prometheus and Loki alert patterns.

Auto-Instrumentation

OpenTelemetry auto-instruments popular frameworks:

from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor

app = FastAPI()
FastAPIInstrumentor.instrument_app(app)  # Auto-trace all HTTP requests

Supported: FastAPI, Flask, Django, Express, Gin, Echo, Nest.js

See: references/opentelemetry-setup.md for framework-specific setup.

Common Patterns

Custom Spans

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("fetch_user_details") as span:
    span.set_attribute("user_id", user_id)
    user = await db.fetch_user(user_id)
    span.set_attribute("user_found", user is not None)

Error Tracking

from opentelemetry.trace import Status, StatusCode

with tracer.start_as_current_span("process_payment") as span:
    try:
        result = process_payment(amount, card_token)
        span.set_status(Status(StatusCode.OK))
    except PaymentError as e:
        span.set_status(Status(StatusCode.ERROR, str(e)))
        span.record_exception(e)
        raise

See: references/trace-context.md for background job tracing and context propagation.

Validation and Testing

# Test log-trace correlation
# 1. Make request to your app
# 2. Copy trace_id from logs
# 3. Query in Grafana: {job="myapp"} |= "trace_id="

# Validate metrics
python scripts/validate_metrics.py

Integration with Other Skills

• Dashboards: Embed Grafana panels, query Prometheus metrics
• Feedback: Alert routing (Slack, PagerDuty), notification UI
• Data-Viz: Time-series charts, trace waterfall, latency heatmaps

See: examples/fastapi-otel/ for complete integration.

Progressive Disclosure

Setup Guides:

• references/opentelemetry-setup.md - SDK installation (Python, Rust, Go, TypeScript)
• references/structured-logging.md - structlog, tracing, slog, pino configuration
• references/lgtm-stack.md - LGTM deployment (Docker, Kubernetes)
• references/trace-context.md - Log-trace correlation patterns
• references/alerting-rules.md - Prometheus and Loki alert templates

Examples:

• examples/fastapi-otel/ - FastAPI + OpenTelemetry + LGTM
• examples/axum-tracing/ - Rust Axum + tracing + LGTM
• examples/lgtm-docker-compose/ - Production-ready LGTM stack

Scripts:

• scripts/setup_otel.py - Bootstrap OpenTelemetry SDK
• scripts/generate_dashboards.py - Generate Grafana dashboards
• scripts/validate_metrics.py - Validate metric naming

Key Principles

1. OpenTelemetry is THE standard - Use OTel SDK, not vendor-specific SDKs
2. Auto-instrumentation first - Prefer auto over manual spans
3. Always correlate logs and traces - Inject traceid/spanid into every log
4. Use structured logging - JSON format, consistent field names
5. LGTM stack for self-hosting - Production-ready open-source stack

Common Pitfalls

Don't:

• Use vendor-specific SDKs (use OpenTelemetry)
• Log without traceid/spanid context
• Manually instrument what auto-instrumentation covers
• Mix logging libraries (pick one: structlog, tracing, slog, pino)

Do:

• Start with auto-instrumentation
• Add manual spans only for business-critical operations
• Use semantic conventions for span attributes
• Export to OTLP (gRPC preferred over HTTP)
• Test locally with LGTM docker-compose before production

Success Metrics

1. 100% of logs include trace_id when in request context
2. Mean time to resolution (MTTR) decreases by >50%
3. Developers use Grafana as first debugging tool
4. 80%+ of telemetry from auto-instrumentation
5. Alert noise < 5% false positives

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: ancoleman
• Source: ancoleman/ai-design-components
• License: MIT

Install and usage instructions live in the source repository linked above.