Kelly Agent Observability

Agent Fleet Observability Desk

Overview

Use this skill as a local operator surface over a MOCK fleet of LLM agents running behind a shared AI gateway for a generic organization. It aggregates per-agent call volume, latency (p50/p95), error/failure rate, and cost into one file-backed App-in-Skill dashboard with a Fleet Overview, a sortable Agent Health table, and a Trace Detail view that highlights exactly where a chain of tool calls broke.

Default interaction mode: App UI. Unless the user explicitly asks for chat-only handling, seed mock telemetry if missing, start/reuse the local app with app/start.sh, and give the actual local URL. Use chat-only mode only when the user says "纯聊天", "chat only", "no UI", or similar.

This is a demo/reference dashboard: all data is generated locally by scripts/generate_fleet_data.ts. There is no real gateway, no real agents, and no external network calls anywhere in this skill.

App UI Screenshots

Fleet OverviewTotal calls, total cost, degraded/critical/healthy agent counts, and a per-agent sparkline card grid. Agent HealthSortable table with call volume, p50/p95 latency, error rate, cost, and a status badge per agent.

Trace DetailOrdered step timeline for one trace; the step where the chain broke is visually flagged. Chinese UIFull zh-CN chrome via the language toggle or lang=zh.

Boundary

• Local, read-only mock dashboard. The app reads and writes local files only.
• The only mutating endpoint is POST /api/handoffs, which appends a

human-in-the-loop acknowledgement or investigation note to a local app/.data/handoffs.jsonl file. It never calls any remote system, agent gateway, or third-party API.

• Never reference any real company, brand, or production system. Agent names

are generic archetypes (Booking Assistant, Support Triage, etc.) for a generic organization ("your organization", "the fleet").

Local App

Start the dashboard with:

skills/kelly-agent-observability/app/start.sh

First run installs hono and @hono/node-server, then seeds mock telemetry into app/.data/fleet.json if it doesn't already exist. The frontend is zero-build vanilla JS/HTML/CSS. The app uses local HTTP on 127.0.0.1, preferring port 3100 through 4100, or KELLY_AGENT_OBS_UI_PORT when set.

To regenerate mock telemetry at any time:

node skills/kelly-agent-observability/scripts/generate_fleet_data.ts

Demo Mode

• ?demo=1 opens a deterministic, fully offline mock fleet (8 agents, 48h of

hourly buckets, ~16 traces per agent) for documentation and screenshots, without touching app/.data/fleet.json.

• ?demo=agents and ?demo=trace select named mock scenes (agent health table,

a broken trace).

• lang=en or lang=zh forces UI chrome language for screenshots.
• Demo API responses never read or write local files.

UI language: support English and Chinese chrome with Auto default.

Data Model

Read references/fleet-schema.md before editing the app, scripts, or the generator. Primary local files:

• app/.data/fleet.json: generated mock telemetry (agents, per-agent metrics,

hourly buckets, traces). Regenerated by scripts/generate_fleet_data.ts.

• app/.data/handoffs.jsonl: append-only human-in-the-loop handoff log written

by POST /api/handoffs.

Views

• #/overview: total calls (24h), total cost (today), degraded/critical/

healthy agent counts, and a per-agent card grid with a call-volume sparkline.

• #/agents: sortable Agent Health table (call volume, p50/p95 latency, error

rate, cost, status badge).

• #/agents/: agent detail — metrics, sparkline, recent traces, and

a handoff form (acknowledge / needs investigation).

• #/traces/: trace detail — ordered step timeline with the

chain-break step visually flagged, and a handoff form.

• #/handoffs: history of recorded handoffs.
• #/settings: data provider and status-threshold summary.

Safety

• Never invent a real incident or real company data; this is illustrative mock

telemetry only.

• Keep all writes local and file-based (app/.data/handoffs.jsonl). No

external network calls anywhere in this skill.

• Redact nothing sensitive is collected here — there are no credentials in

this skill.

Execution reports

Re-read the active provider's decisions immediately before any approved execution. Record each concrete operation, target, status, timestamp, and error in the provider-backed execution report; keep app actions local-only.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: mr-kelly
• Source: mr-kelly/skills
• License: MIT
• Homepage: https://mr-kelly.github.io/skills/

Install and usage instructions live in the source repository linked above.