# Ai Observability And Debugging

> Use when building the observability stack for AI features in a multi-tenant SaaS — prompt/response tracing, semantic logging, replay tooling, "show me why this answer", per-stage latency/cost breakdown, ticket→trace tie-back, and dashboards that answer the operational questions (which tenant, which feature, which prompt version, which model).

- **Type:** Skill
- **Install:** `agentstack add skill-peterbamuhigire-skills-web-dev-ai-observability-and-debugging`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [peterbamuhigire](https://agentstack.voostack.com/s/peterbamuhigire)
- **Installs:** 0
- **Category:** [Databases](https://agentstack.voostack.com/c/databases)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [peterbamuhigire](https://github.com/peterbamuhigire)
- **Source:** https://github.com/peterbamuhigire/skills-web-dev/tree/main/skills/ai/ai-observability-and-debugging
- **Website:** https://techguypeter.com

## Install

```sh
agentstack add skill-peterbamuhigire-skills-web-dev-ai-observability-and-debugging
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# AI Observability and Debugging
Acknowledgement: Shared by Peter Bamuhigire, techguypeter.com, +256 784 464178.

## Use When

- A support ticket arrives saying "this answer is wrong" and you need to pull up the exact prompt, model, retrieval state, and reasoning in  --window  --output `) that snapshots state-at-window-end (prompt/model/index/price registries), pulls a stratified sample of failing traces, and generates reproduce scripts. See `ai-incident-evidence-capture/references/evidence-bundle-spec.md` for the schema and `ai-incident-evidence-capture/references/reproduce-script-template.md` for the script generator. The replay tooling must support a "show me everything from  for  on " mode usable by an on-call engineer at T+5 of an incident.

## §1 AI Trace Schema

A standard `ai.*` namespace under OpenTelemetry semantic conventions. See `references/trace-schema.md` for the full schema. Summary:

- Root span: `ai.gateway.request`
- Children: `ai.entitlement`, `ai.kill_switch`, `ai.rate_limit`, `ai.caps`, `ai.prompt.render`, `ai.safety.in`, `ai.retrieval`, `ai.provider.call`, `ai.safety.out`, `ai.cost`, `ai.audit`.
- Attributes on root: `ai.tenant_id`, `ai.feature`, `ai.prompt_id`, `ai.prompt_version`, `ai.model_used`, `ai.region`, `ai.tokens_in`, `ai.tokens_out`, `ai.usd_cost`, `ai.fallback_used`.

## §2 Per-Stage Spans

Each pipeline stage produces a span with:
- `service.name = "llm-gateway"`
- `ai.stage = ""`
- `ai.outcome = "ok|denied|error"`
- duration

```python
with tracer.start_as_current_span("ai.entitlement") as span:
    span.set_attribute("ai.stage", "entitlement")
    span.set_attribute("ai.tenant_id", ctx.tenant_id)
    span.set_attribute("ai.feature", ctx.feature)
    try:
        await entitlements.check(ctx)
        span.set_attribute("ai.outcome", "ok")
    except NotEntitled as e:
        span.set_attribute("ai.outcome", "denied")
        span.set_attribute("ai.denial_key", e.key)
        raise
```

Traces export to OTel; sampled at 100% for AI traces by default (volumes are low enough; signal is critical). Reduce sampling once volume scales.

## §3 Replay

Given a `request_id`, the replay tool:

1. Pulls the full audit row from `ai_requests` + S3 payload.
2. Pulls the resolved binding at the time of the request (snapshot in payload).
3. Reconstructs the exact gateway input.
4. Runs the gateway pipeline in **replay mode**:
   - Real provider call OR replayed provider response (from payload).
   - Same prompt version, same KB partition, same model.
   - Same safety classifier version.
5. Diffs the response with the original; shows differences.

Two modes:
- `--with-provider`: re-issue to provider (costs money; useful for "is this still broken?").
- `--from-cache`: use stored provider response (free; useful for analysing pipeline changes).

CLI: `replay --request-id ai_req_01HXY... --mode from-cache`

Replay is **read-only**; never charges credits, never affects metrics.

## §4 "Show Me Why" Surfaces

Two audiences:

**Back-office operator view:** full trace, payload, retrieval chunks with scores, safety findings, judge-LLM score (if sampled). Filterable by tenant, feature, time, error class. Drill from a customer ticket.

**Tenant admin view (optional, premium):** for each AI answer, an "explain" affordance that shows:
- The retrieval chunks used (with snippets, scores, sources).
- The prompt template id and version.
- The model used.
- The grounding score and citations.

Hide:
- The platform system prompt.
- Any internal classifier scores.
- Cost.

Make tenant-facing explanations always derived from the audit log; never re-generated by the model (that would be the model rationalising, not explaining).

## §5 Ticket → Trace Tie-Back

Every AI feature surfaces a copyable `request_id` (often the last 8 chars) in the UI as a small badge or in a "report issue" flow. The support tool accepts the id and:

1. Resolves the audit row.
2. Renders the operator view.
3. Links to the trace.
4. Links to the replay tool.

For high-volume features, also expose a "feedback?" thumbs-up/down per response, recorded with the `request_id`.

## §6 Dashboards

Standard set per feature:

| Dashboard | Panels |
|---|---|
| Feature ops | QPS, p50/p95/p99 latency by stage, error rate by stage, fallback ratio, abstain rate |
| Cost | USD per hour, per tenant top-20, per model, per feature share |
| Quality | faithfulness, correctness, citation density, judge agreement (rolling) |
| Safety | injection detect rate, output blocks, tool denials, jailbreak suspicions |
| Tenant view | per-tenant requests, cost, errors, denials, top features |

Each panel filterable by time, tenant, region, model, prompt version.

## §7 Anti-Patterns

- Mixing AI traces with general request traces with no namespace — discoverability rot.
- Storing payloads in trace attributes (massive cardinality, cost). Use S3 with a key in attributes.
- Logging full prompt to stdout (printed to log aggregator with no redaction). PII leak.
- No way to see retrieval chunks in the operator UI. Half the diagnosis missing.
- Dashboards built once by a single engineer; no shared template. Drift across features.
- Replay tool that uses the current price table to compute cost on historical replays. Misleading.
- Sampling at < 10% on AI requests. AI volume is usually small enough to keep 100%.

## §8 Agent Trace Patterns

Single-request AI observability is insufficient for agentic features. An agent task is **one trace** spanning many LLM calls and tool calls, often running for minutes or hours. The trace schema, replay tooling, and task-level dashboards live in `ai-agent-observability-and-replay`.

Key differences from request-level traces:

| Dimension | Request | Agent task |
|---|---|---|
| Trace lifetime | seconds | seconds to days |
| Spans per trace | 5-20 | 50-500+ |
| Replay | re-call with same inputs | re-run loop with mocked tool I/O |
| Failure modes | latency, content, cost | + step-loop, hallucinated tool args, off-script irreversibles, deadlock |
| Dashboards | per-request metrics | per-task metrics: success rate, intervention rate, step efficiency |

Required schema additions when agents are in scope — root span `agent.task` with attributes for `agent.task.id`, `agent.feature`, `agent.prompt.version`, `agent.tool_set.version`, `agent.task.steps.used/budget`, `agent.task.usd.total`, `agent.task.terminal.state`. Per-step child spans `agent.step.N`. Per-LLM-call and per-tool-call sub-spans. Special spans: `agent.handoff`, `agent.approval`. Full schema: `ai-agent-observability-and-replay/references/trace-schema-agent.md`.

Required dashboards: per-feature task success rate, intervention rate, irreversible-action rate, step efficiency, median cost per completed task; per-tenant agent task volume, agent cost share, wasted-spend share; top-10 tasks by cost in last hour; failure-mode distribution.

Required replay surface: per-task replay re-running a recorded task with candidate prompt/model/tool version; side-by-side step diff; never call live tools in replay.

## §9 Read Next

- `ai-agent-observability-and-replay` — the **agent-specific** complement with full trace schema, replay tooling, task viewer, and "what would the agent do differently" debugger.
- `ai-agent-runtime-architecture` — emits the task-level trace events.
- `ai-agent-eval` — consumes traces for replay-based eval.
- `ai-on-saas-architecture` — control-plane positioning.
- `ai-model-gateway` — instrumented from.
- `ai-eval-harness` — surfaces quality signal to dashboards.
- `ai-prompt-injection-and-tenant-safety` — safety signals to dashboards.
- `observability-monitoring` — broader observability.
- `saas-admin-backoffice-tooling` — operator UI.
## Consolidated Child References

- Load [references/routing.md](references/routing.md) to map retired AI child skill slugs to their reference modules.

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [peterbamuhigire](https://github.com/peterbamuhigire)
- **Source:** [peterbamuhigire/skills-web-dev](https://github.com/peterbamuhigire/skills-web-dev)
- **License:** MIT
- **Homepage:** https://techguypeter.com

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-peterbamuhigire-skills-web-dev-ai-observability-and-debugging
- Seller: https://agentstack.voostack.com/s/peterbamuhigire
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.