Ai Observability And Debugging

AI Observability and Debugging

Acknowledgement: Shared by Peter Bamuhigire, techguypeter.com, +256 784 464178.

Use When

• A support ticket arrives saying "this answer is wrong" and you need to pull up the exact prompt, model, retrieval state, and reasoning in --window --output ) that snapshots state-at-window-end (prompt/model/index/price registries), pulls a stratified sample of failing traces, and generates reproduce scripts. See ai-incident-evidence-capture/references/evidence-bundle-spec.md for the schema and ai-incident-evidence-capture/references/reproduce-script-template.md` for the script generator. The replay tooling must support a "show me everything from for on " mode usable by an on-call engineer at T+5 of an incident.

§1 AI Trace Schema

A standard ai.* namespace under OpenTelemetry semantic conventions. See references/trace-schema.md for the full schema. Summary:

• Root span: ai.gateway.request
• Children: ai.entitlement, ai.kill_switch, ai.rate_limit, ai.caps, ai.prompt.render, ai.safety.in, ai.retrieval, ai.provider.call, ai.safety.out, ai.cost, ai.audit.
• Attributes on root: ai.tenant_id, ai.feature, ai.prompt_id, ai.prompt_version, ai.model_used, ai.region, ai.tokens_in, ai.tokens_out, ai.usd_cost, ai.fallback_used.

§2 Per-Stage Spans

Each pipeline stage produces a span with:

• service.name = "llm-gateway"
• ai.stage = ""
• ai.outcome = "ok|denied|error"
• duration

with tracer.start_as_current_span("ai.entitlement") as span:
    span.set_attribute("ai.stage", "entitlement")
    span.set_attribute("ai.tenant_id", ctx.tenant_id)
    span.set_attribute("ai.feature", ctx.feature)
    try:
        await entitlements.check(ctx)
        span.set_attribute("ai.outcome", "ok")
    except NotEntitled as e:
        span.set_attribute("ai.outcome", "denied")
        span.set_attribute("ai.denial_key", e.key)
        raise

Traces export to OTel; sampled at 100% for AI traces by default (volumes are low enough; signal is critical). Reduce sampling once volume scales.

§3 Replay

Given a request_id, the replay tool:

1. Pulls the full audit row from ai_requests + S3 payload.
2. Pulls the resolved binding at the time of the request (snapshot in payload).
3. Reconstructs the exact gateway input.
4. Runs the gateway pipeline in replay mode:

• Real provider call OR replayed provider response (from payload).
• Same prompt version, same KB partition, same model.
• Same safety classifier version.

1. Diffs the response with the original; shows differences.

Two modes:

• --with-provider: re-issue to provider (costs money; useful for "is this still broken?").
• --from-cache: use stored provider response (free; useful for analysing pipeline changes).

CLI: replay --request-id ai_req_01HXY... --mode from-cache

Replay is read-only; never charges credits, never affects metrics.

§4 "Show Me Why" Surfaces

Two audiences:

Back-office operator view: full trace, payload, retrieval chunks with scores, safety findings, judge-LLM score (if sampled). Filterable by tenant, feature, time, error class. Drill from a customer ticket.

Tenant admin view (optional, premium): for each AI answer, an "explain" affordance that shows:

• The retrieval chunks used (with snippets, scores, sources).
• The prompt template id and version.
• The model used.
• The grounding score and citations.

Hide:

• The platform system prompt.
• Any internal classifier scores.
• Cost.

Make tenant-facing explanations always derived from the audit log; never re-generated by the model (that would be the model rationalising, not explaining).

§5 Ticket → Trace Tie-Back

Every AI feature surfaces a copyable request_id (often the last 8 chars) in the UI as a small badge or in a "report issue" flow. The support tool accepts the id and:

1. Resolves the audit row.
2. Renders the operator view.
3. Links to the trace.
4. Links to the replay tool.

For high-volume features, also expose a "feedback?" thumbs-up/down per response, recorded with the request_id.

§6 Dashboards

Standard set per feature:

| Dashboard | Panels | |---|---| | Feature ops | QPS, p50/p95/p99 latency by stage, error rate by stage, fallback ratio, abstain rate | | Cost | USD per hour, per tenant top-20, per model, per feature share | | Quality | faithfulness, correctness, citation density, judge agreement (rolling) | | Safety | injection detect rate, output blocks, tool denials, jailbreak suspicions | | Tenant view | per-tenant requests, cost, errors, denials, top features |

Each panel filterable by time, tenant, region, model, prompt version.

§7 Anti-Patterns

• Mixing AI traces with general request traces with no namespace — discoverability rot.
• Storing payloads in trace attributes (massive cardinality, cost). Use S3 with a key in attributes.
• Logging full prompt to stdout (printed to log aggregator with no redaction). PII leak.
• No way to see retrieval chunks in the operator UI. Half the diagnosis missing.
• Dashboards built once by a single engineer; no shared template. Drift across features.
• Replay tool that uses the current price table to compute cost on historical replays. Misleading.
• Sampling at < 10% on AI requests. AI volume is usually small enough to keep 100%.

§8 Agent Trace Patterns

Single-request AI observability is insufficient for agentic features. An agent task is one trace spanning many LLM calls and tool calls, often running for minutes or hours. The trace schema, replay tooling, and task-level dashboards live in ai-agent-observability-and-replay.

Key differences from request-level traces:

| Dimension | Request | Agent task | |---|---|---| | Trace lifetime | seconds | seconds to days | | Spans per trace | 5-20 | 50-500+ | | Replay | re-call with same inputs | re-run loop with mocked tool I/O | | Failure modes | latency, content, cost | + step-loop, hallucinated tool args, off-script irreversibles, deadlock | | Dashboards | per-request metrics | per-task metrics: success rate, intervention rate, step efficiency |

Required schema additions when agents are in scope — root span agent.task with attributes for agent.task.id, agent.feature, agent.prompt.version, agent.tool_set.version, agent.task.steps.used/budget, agent.task.usd.total, agent.task.terminal.state. Per-step child spans agent.step.N. Per-LLM-call and per-tool-call sub-spans. Special spans: agent.handoff, agent.approval. Full schema: ai-agent-observability-and-replay/references/trace-schema-agent.md.

Required dashboards: per-feature task success rate, intervention rate, irreversible-action rate, step efficiency, median cost per completed task; per-tenant agent task volume, agent cost share, wasted-spend share; top-10 tasks by cost in last hour; failure-mode distribution.

Required replay surface: per-task replay re-running a recorded task with candidate prompt/model/tool version; side-by-side step diff; never call live tools in replay.

§9 Read Next

• ai-agent-observability-and-replay — the agent-specific complement with full trace schema, replay tooling, task viewer, and "what would the agent do differently" debugger.
• ai-agent-runtime-architecture — emits the task-level trace events.
• ai-agent-eval — consumes traces for replay-based eval.
• ai-on-saas-architecture — control-plane positioning.
• ai-model-gateway — instrumented from.
• ai-eval-harness — surfaces quality signal to dashboards.
• ai-prompt-injection-and-tenant-safety — safety signals to dashboards.
• observability-monitoring — broader observability.
• saas-admin-backoffice-tooling — operator UI.

Consolidated Child References

• Load [references/routing.md](references/routing.md) to map retired AI child skill slugs to their reference modules.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: peterbamuhigire
• Source: peterbamuhigire/skills-web-dev
• License: MIT
• Homepage: https://techguypeter.com

Install and usage instructions live in the source repository linked above.