Java Obs Agent

java-obs-agent

[](LICENSE) [](README.zh-CN.md)

A framework-agnostic Java observability sidecar agent — drop a single -javaagent jar onto any Java (Servlet) service, then chat with it (via a built-in web UI, an MCP client like Claude Code, or curl) to understand the service's live state: request volume, latency, slow calls, errors, logs, thread dumps, method-level traces.

It's the thing between Arthas (in-JVM diagnostics, no AI/aggregation) and SkyWalking (APM with distributed traces, no conversational agent) — plus the Datadog Bits AI pattern (chat-with-your-service), but self-hosted and attachable to a single JVM.

> Built and validated end-to-end on a real Spring Boot service (the interview-guide project): attach → chat → "which route is slowest?" → agent calls its tools → answers with real metrics, then trace_request to pin the slow method in the call tree.

---

What it does

Attach the agent to a running Java service (no restart needed — dynamic attach like Arthas). The agent runs inside the target JVM:

• Captures per-route request metrics (count, p50/p95/p99, error rate), slow calls, errors — into in-JVM ring buffers (zero external storage).
• Captures recent logs (Logback appender) + thread snapshots (JMX).
• Exposes a MCP server (JSON-RPC over HTTP) — any MCP client (Claude Code, Cursor) can chat with the service.
• Exposes a built-in chat web UI (/chat) — an OpenAI-compatible LLM (BYO API key) orchestrates the tools to answer your questions in natural language.
• On-demand method-level diagnostics: watch_method (capture a method's invocations) and trace_request (full method call-tree, Arthas-trace style).
• Leaves a Source SPI seam so a future SkyWalking/Tempo backend plugs in without changing tools (Phase 3).

Quick start

1. Build

cd 
JAVA_HOME= ./gradlew :obs-agent-core:shadowJar

Produces obs-agent-core/build/libs/obs-agent-core-0.1.0-all.jar (shaded + relocated ByteBuddy/Jackson; slf4j uses the host's). Use the -all jar (has the manifest + deps); not the plain one.

> JDK 21 (GraalVM 21.0.11) builds Java 17 bytecode. The system default java is JDK 8 — always set JAVA_HOME to JDK 21 for ./gradlew.

2. Attach to a running service (dynamic, no restart)

# find the service PID (e.g. on port 8080)
PID=$(netstat -ano | grep LISTENING | grep ":8080" | head -1 | awk '{print $NF}')

# attach the agent into the running JVM (Arthas-style)
JAVA_HOME=
JAR="/obs-agent-core/build/libs/obs-agent-core-0.1.0-all.jar"
ARGS='port=9911,token=tok,enableInstrumentation=true,chat.apiKey=,chat.base_url=,chat.model='
"$JAVA_HOME/bin/java" --add-modules jdk.attach -cp scripts Attach "$PID" "$JAR" "$ARGS"

The agent starts an MCP server + (optional) chat UI on port 9911 inside the target JVM. It stays until the JVM restarts.

> The Attach helper is a one-liner using com.sun.tools.attach.VirtualMachine.loadAgent (the agentmain entry point). It ships at scripts/Attach.java. To use it, compile once: "$JAVA_HOME/bin/javac" --add-modules jdk.attach -d scripts scripts/Attach.java.

2b. Or attach at JVM startup (-javaagent, persistent across restarts)

java -javaagent:/obs-agent-core/build/libs/obs-agent-core-0.1.0-all.jar=port=9911,token=tok,enableInstrumentation=true -jar your-app.jar

Use this if you want the agent resident from boot (survives restarts). Dynamic attach (above) is for ad-hoc diagnosis without restart.

3. Use it

Chat web UI (browser): open http://localhost:9911/chat?token=tok — a chat box; type questions, the LLM calls the tools and answers.

Claude Code (MCP client):

claude mcp add --transport http obs-agent http://localhost:9911/mcp --header "Authorization: Bearer tok"
# then chat: "which route is slowest? any recent errors?"

curl (manual tool call):

curl -X POST http://localhost:9911/mcp \
  -H "Authorization: Bearer tok" -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"recent_metrics","arguments":{"route":"/api/x"}}}'

> All endpoints require Authorization: Bearer . The chat UI page also accepts ?token= (browsers can't send headers on URL navigation).

---

Configuration

All config is in the agent args (comma-separated key=value), optionally overridden by env vars (OBSAGENT_PORT, OBSAGENT_HOST, OBSAGENT_TOKEN).

| Key | Default | Description | |---|---|---| | port | 9000 | MCP + chat HTTP port | | host | 127.0.0.1 | Network interface to bind. Defaults to loopback (not exposed). Set 0.0.0.0 only behind a firewall/reverse proxy and with token set. | | token | (none → no auth) | MCP/chat Bearer token. Set it if the port is exposed beyond localhost. | | window | 300s | Metrics/log retention window | | bucket | 1s | Metrics time bucket | | slowThreshold | 500ms | Slow-call threshold | | logBuffer | 2000 | Log ring buffer size | | enableInstrumentation | false | Master switch for watch_method/trace_request (instrumentation tier) | | watchLimit | 10 | Max captures per watch_method | | watchTtl | 60s | watch_method auto-uninstall TTL | | scrubPatterns | (none) | Regexes (semicolon-separated) to scrub from tool results (tokens/PII) | | chat.apiKey | (none → chat off) | LLM API key. Set to enable the built-in chat. | | chat.base_url | — | OpenAI-compatible base URL (e.g. https://api.openai.com/v1, https://open.bigmodel.cn/api/paas/v4, https://ark.cn-beijing.volces.com/api/coding/v3) | | chat.model | — | Model name (e.g. gpt-4o-mini, glm-4, glm-5.2) |

When chat.apiKey is unset, only MCP is exposed (chat UI returns 404).

---

Tools

Read-only (always available):

| Tool | What it does | |---|---| | get_routes | Known routes + health (count, error rate, p95) | | recent_metrics(route?, window_seconds) | Per-route count / p50/p95/p99 / error rate / qps | | slow_calls(top) | Slowest recent requests (with traceId if host sets MDC) | | recent_errors(top) | Recent error requests | | search_logs(query, level?, traceId?) | Search the in-JVM log ring (host Logback) | | thread_dump(top, by=cpu\|blocked) | Top CPU threads + deadlocks |

Instrumentation tier (gated by enableInstrumentation=true, default off):

| Tool | What it does | |---|---| | watch_method(class, method, limit?, ttl_seconds?) | Start capturing a method's invocations (args/return/duration/exception). Returns immediately (two-phase). | | watch_results(class, method) | Read the captures from a prior watch_method. | | trace_request(class_pattern, ttl_seconds?, limit?) | Start a method call-tree trace (Arthas-trace) for classes matching the regex. Returns immediately. | | trace_results() | Read the call trees from a prior trace_request. |

Two-phase design: watch_method/trace_request install + return immediately; the trace/watch stays active for ttl_seconds. Generate traffic (hit the endpoint, or rely on frontend traffic), then call watch_results/trace_results to read. This is critical — a single blocking call would prevent the caller (LLM) from generating traffic during the window (→ empty results).

trace_request usage

• class_pattern is a regex. Use the FQCN for specific classes.
• To see cross-class call trees (controller → service), trace both classes with |: e.g. class_pattern="interview.guide.modules.x.KnowledgeBaseController|interview.guide.modules.x.service.KnowledgeBaseListService".
• Repeated trace_request calls reset the prior trace before installing the new one (no advice stacking).
• Example flow (via chat or MCP):

1. trace_request("...KnowledgeBaseController|...KnowledgeBaseListService", ttl_seconds=15)
2. hit the endpoint a few times
3. trace_results() → tree: controller.getStatistics (335ms) → service.getStatistics (333ms)

---

Architecture

┌──────────────────────────────────────────────────────────┐
│  LLM clients (BYO): chat UI (/chat) / Claude Code / curl  │
└──────────────────┬───────────────────────────────────────┘
                   │ MCP (JSON-RPC over HTTP) + /chat (POST), token auth
┌──────────────────▼───────────────────────────────────────┐
│  obs-agent-core  (inside the target JVM, shaded jar)      │
│  ┌───────────┐  ┌────────────┐  ┌─────────────────────┐   │
│  │ MCP server│  │ ChatEndpoint│  │  ToolRegistry       │   │
│  │ (JdkHttp) │  │ (LlmClient, │  │ (10 tools)          │   │
│  │           │  │  agent loop)│  │                     │   │
│  └───────────┘  └────────────┘  └──────────┬──────────┘   │
│  ┌──────────────────────────────────────────▼──────────┐ │
│  │ Source SPI (extension seam):                        │ │
│  │ MetricsSource / LogSource / TraceSource /            │ │
│  │ TopologySource  (+ capability flags)                 │ │
│  └──────────────────────────┬───────────────────────────┘ │
│  ┌──────────────────────────▼──────────────────────────┐│
│  │ Phase 1 in-JVM impls: ring buffers / Logback / JMX   ││
│  └──────────────────────────────────────────────────────┘│
│  ┌──────────────────────────────────────────────────────┐│
│  │ ByteBuddy instrumentation (shaded + isolated):        ││
│  │ - ServletMetricsInstrumentation (javax+jakarta)       ││
│  │ - MethodInstrumenter (watch)  - MethodTracer (trace)   ││
│  └──────────────────────────────────────────────────────┘│
└────────────────────────────────────────────┬──────────────┘
              (future) ┌──────────────────────▼──────────────┐
                       │ Phase 3: SkyWalkingTraceSource etc.  │
                       │ (query OAP, zero-change to tools)    │
                       └──────────────────────────────────────┘

Key design points:

• Single shaded jar — ByteBuddy/Jackson relocated to io.obsagent.shaded.*; slf4j is compileOnly (uses the host's, so the agent can read host Logback/MDC). No host-classpath pollution.
• No new deps for chat — uses JDK java.net.http.HttpClient (OpenAI-compatible).
• Advice never throws into user code — all ByteBuddy advice is wrapped in try/catch + JUL logging.
• Advice-accessed members are public static — ByteBuddy inlines advice into target classes in other packages; package-private fields/methods would IllegalAccessError (a real gotcha — see Troubleshooting).
• agentmain — enables dynamic attach to a running JVM (not just -javaagent at startup).

---

Roadmap

• ✅ Phase 1 — core agent: servlet metrics + ring buffers + Source SPI + in-JVM sources + MCP server + read-only tools + scrubber/guard + shaded jar + e2e smoke test.
• ✅ Phase 2 — watch_method (method capture) + trace_request (call-tree) + validation demo.
• ✅ Built-in chat — LlmClient (OpenAI-compatible) + ChatEndpoint (agent loop) + web UI at /chat.
• ✅ Dynamic attach (agentmain) + two-phase trace/watch + trace reset (no stacking) + cross-class call trees.
• ⏳ Phase 3 (deferred) — SkyWalkingTraceSource/SkyWalkingTopologySource (query OAP) → distributed_trace/service_topology tools → cross-service business-chain root cause (order→pay). The Source SPI is already in place.
• ⏳ Optional tail — Spring Boot adapter (route templates/Actuator/Micrometer traceId); streaming chat (token-level SSE); obs-agent-cli; reactive/WebFlux; Java 8 bytecode broadening.

---

Known limitations

• trace_request is pattern-based, not traceId-based — the host must set MDC traceId (e.g. Micrometer Tracing) for slow_calls/search_logs to correlate by traceId. Without it, those fields are null (route+latency diagnosis still works).
• Broad .* patterns capture the entry method but may not show cross-class callees as children — trace the specific classes (with |) for cross-class trees.
• Chat is non-streaming (POST /chat/completions, stream=false) — returns the full answer after the tool-calling loop. Live token streaming is a follow-up.
• Servlet-only (blocking) — reactive/WebFlux not yet instrumented.
• Java 17 bytecode — broadening to Java 8 (attach to older JVMs) is a follow-up.
• Agent can't be hot-updated in a running JVM — re-attaching a new jar doesn't redefine already-loaded classes (system classloader caches them). To pick up code changes, restart the target JVM + re-attach (or use -javaagent at startup).
• watch_method/trace_request default OFF — set enableInstrumentation=true to use them.

---

Troubleshooting

MCP client (Claude Code) fails to connect with "expected object, received undefined" for capabilities/serverInfo — the initialize result must include both (MCP spec). Fixed (returns {"protocolVersion","capabilities":{"tools":{}},"serverInfo":{"name":"obs-agent","version":"0.1.0"}}).

MCP tools return "content expected array, received object" / content blocks fail validation — tools/call result must be {"content":[{"type":"text","text":""}],"isError":false} (array of content blocks, each with a type). Fixed.

search_logs returns "no log source" / traceId always null — slf4j must NOT be shaded (so the agent uses the host's slf4j to reach Logback + MDC). slf4j is compileOnly + not relocated. If you see this, check the shadow config.

Browser opens /chat → {"code":-32001,"message":"unauthorized"} — browsers can't send Authorization headers on URL navigation. Open http://localhost:9911/chat?token=tok (query param accepted).

trace_request returns empty {"trees":[]} — (1) it's two-phase: call trace_request (install), generate traffic, then trace_results (read). The install call returns immediately, it does NOT capture. (2) Make sure enableInstrumentation=true. (3) Verify the class_pattern regex matches (use the FQCN; . matches any char, * is a quantifier).

trace_request shows duplicate/nested trees — fixed: repeated trace_request now reset()s the prior transformer before installing. If you still see it, you're on an old jar (rebuild + re-attach).

ByteBuddy advice silently does nothing (empty captures, no error) — the advice accesses package-private members from a cross-package target → IllegalAccessError swallowed by the try/catch → returns null. Advice logic must be in public static methods (e.g. onEnter/onExit) so the inlined advice only calls public methods. The unit test passes (same-package test class) but live cross-package targets fail — always verify on a fresh JVM with a real cross-package target.

Build fails with JDK issues — system java is JDK 8; the project needs JDK 21. Always run ./gradlew with JAVA_HOME=.

Re-attaching a second agent doesn't apply new code — the first attach's classes are already loaded by the system classloader; loadAgent won't redefine them. Restart the target JVM + attach once.

---

Development

# build
JAVA_HOME="..." ./gradlew :obs-agent-core:build
# tests
JAVA_HOME="..." ./gradlew :obs-agent-core:test
# e2e smoke (real -javaagent attach + MCP)
JAVA_HOME="..." ./gradlew :obs-agent-core:test --tests 'io.obsagent.EndToEndSmokeTest' -Pe2e=true
# shaded agent jar
JAVA_HOME="..." ./gradlew :obs-agent-core:shadowJar

Layout:

• obs-agent-core/src/main/java/io/obsagent/ — AgentPremain, config/, data/ (ring buffers + model records), source/ (SPI + injvm/ impls), instr/ (ByteBuddy: servlet/watch/trace), mcp/ (JdkHttpMcpServer), tools/ (ReadOnlyTools + InstrumentingTools), safety/ (Scrubber + Guard), chat/ (LlmClient + ChatEndpoint + ChatPage).
• docs/superpowers/specs/ — design spec.
• docs/superpowers/plans/ — Phase 1 / Phase 2 / chat+trace implementation plans.
• .superpowers/sdd/progress.md — SDD execution ledger (gitignored).

---

How it was built

Designed via brainstorming → spec → implementation plans → subagent-driven development (12 Phase-1 tasks + 4 Phase-2/chat-trace tasks),

…

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: twz007
• Source: twz007/java-obs-agent
• License: Apache-2.0

Install and usage instructions live in the source repository linked above.