Ratel Langsmith Integrate

/ratel-langsmith-integrate — wire LangSmith coverage and dashboards for an agent

Mount LangSmith on a customer's codebase the way the Ratel team would, and spec the dashboards that prove it earned its place. One skill, two halves: instrumentation wiring (where tracing belongs, what to name it, how to set it up) and the dashboard build-spec (which charts to build on the Monitor and custom dashboards, with concrete configurations). The plan is the deliverable. Do not edit the agent code, and do not call the LangSmith API — the customer builds dashboards by clicking through the LangSmith UI; that is intentional.

This is the LangSmith branch of the observability funnel. The generic [/ratel-observability-assessment](../ratel-observability-assessment/SKILL.md) decides what to capture and which dashboards matter, vendor-neutrally, and detects the vendor. When that vendor is LangSmith, it routes here to render the plan into concrete LangSmith primitives. Its downstream follow-up, once traces are flowing, is [/ratel-langsmith-analyze](../ratel-langsmith-analyze/SKILL.md), which reads the live data this plan produces.

Philosophy: trace the mental model, not the call graph

A common failure mode is "wrap every function in a run." That produces run trees that match the code's call graph but tell you nothing about what the agent was trying to do. LangSmith traces are most useful when their structure matches the conceptual structure of a turn:

• Trace (root run) = one externally meaningful unit of work (one chat turn, one job, one webhook). Not "one HTTP request" if a request contains multiple agent turns; not "one model call" if a turn contains many.
• Child run = one step the agent took inside that unit. Sub-agent invocations (chain runs), tool calls (tool runs), model calls (llm runs), retrieval steps (retriever runs). Nest them to reflect delegation, not source-file layout.
• Thread = a set of related traces sharing a thread metadata key (session_id, thread_id, or conversation_id). Usually a user conversation, an agent run-id, or a job correlation id.

The full rationale for this lives once in [../ratel-observability-assessment/references/instrumentation-philosophy.md](../ratel-observability-assessment/references/instrumentation-philosophy.md). The LangSmith-specific rendering of the vocabulary — which generic concept maps to which run type and metadata key — lives in [references/langsmith-mapping.md](references/langsmith-mapping.md).

Why two groups of dashboards

The dashboard half of this skill always builds two groups, because partner startups want two different stories from the same data:

1. "Ratel is moving the numbers" (Ratel-value group) — token spend down, retrieval quality up, fewer "tool not found" errors, lower cost per session. These justify the engagement. Included only if Ratel is present or planned.
2. "Our agent is healthy" (agent-health group) — latency percentiles, error rates per tool, abandoned-session rates, feedback-score distributions. Useful regardless of Ratel; they build trust because they help the customer's own engineers find their own bugs.

Ratel-only dashboards feel like a sales pitch; agent-health-only dashboards feel like we forgot why we're there. Include both whenever Ratel is in play. LangSmith's built-in Monitor tab already covers much of the agent-health story per project — don't rebuild what it gives for free (see Step 5).

Workflow

Step 0 — Read the upstream observability assessment (preferred input)

Look for /.ratel/ratel-observability-assessment.md — the deliverable of [/ratel-observability-assessment](../ratel-observability-assessment/SKILL.md). If present, it is the preferred input: read its detected stack, topology, vendor-neutral instrumentation strategy, and recommended-dashboard list. Confirm the detected vendor is LangSmith; if it says Langfuse (or another vendor), stop and tell the user this is the LangSmith skill — they want the matching *-integrate skill instead.

If the file is present, you can skip the heavy detection in Steps 1–2 and jump to rendering: use its stack and topology directly, then go to Step 3.

If the file is absent, do not hard-block. Do your own quick stack + vendor detect (Steps 1–2) and, like a dashboard build needs a vocabulary, ask a few questions rather than refusing:

1. What's the canonical root run name for one chat turn / one job?
2. What env, stack, and agent_version tag values are in use (or planned)?
3. Is Ratel instrumented in any form (gateway, SDK, or planned)?

Proceed with answers in hand. Charts built on guessed vocabulary look right but group wrong.

Step 1 — Detect the stack

(Skip if the upstream assessment already names the stack.) Read manifest files to identify language and framework. Branch into the matching reference for stack-specific patterns.

# TypeScript / Node detection
test -f package.json && cat package.json | jq -r '.dependencies // {}, .devDependencies // {} | keys[]' | sort -u

# Python detection
test -f pyproject.toml && cat pyproject.toml | grep -A 200 '^\[' || true
test -f requirements.txt && cat requirements.txt
test -f uv.lock && head -50 uv.lock

Map dependencies to one of these stack profiles:

| Signal in manifest | Stack | Reference | | --- | --- | --- | | ai, @ai-sdk/* | Vercel AI SDK | [references/stack-vercel-ai-sdk.md](references/stack-vercel-ai-sdk.md) | | @mastra/core, hand-rolled loops calling openai / @anthropic-ai/sdk directly | TypeScript generic | [references/stack-typescript-generic.md](references/stack-typescript-generic.md) | | langsmith + openai / anthropic / langchain / llama_index | Python generic | [references/stack-python-generic.md](references/stack-python-generic.md) | | langgraph, crewai, agno, autogen | Python agentic | [references/stack-python-agentic.md](references/stack-python-agentic.md) |

If signals overlap (e.g., both a LangGraph supervisor and raw OpenAI calls inside), pick the agentic reference as primary and note the mixed-stack callout in the plan.

If you cannot identify any agent surface at all (no LLM client imports, no agent framework, no model calls), use the [honest skip path](#honest-skip-path).

To lock exact current identifiers (SDK names, wrapAISDK availability, env vars), fetch the live docs at authoring time — docs.langchain.com/langsmith (annotate-code, threads, dashboards) or the langchain-ai/langsmith-sdk docs via Context7. Do not invent APIs.

Step 2 — Map the agent's topology

(Skip / reuse if the upstream assessment already has a topology diagram.) Launch one Explore agent (or do it directly for very small repos) to answer four questions, citing file paths:

1. Where does a turn begin? — entry points: an HTTP handler, a CLI verb, a queue consumer, a chat-platform webhook. This is where the thread key (session_id) is sourced.
2. What are the agent units? — supervisor function, sub-agent factories, role-specialised loops. Anything that takes a user message and returns a response. These become root/child run boundaries.
3. Where are tools defined and called? — tool registries (tools: [...]), @tool decorators, MCP server wiring. Each tool needs to surface as a run of type tool.
4. Where do sub-agents hand off to other sub-agents? — supervisor → worker, parallel fan-out, graph node transitions. These are the spots where the thread key and metadata must be re-attached, since LangSmith does not propagate metadata or tags from a parent run to its children.

Capture this as a small topology diagram in the plan (ASCII or Mermaid). It does not need to be exhaustive — it needs to give the customer a single picture they can point at while implementing.

Step 3 — Render the LangSmith naming/mapping table

Read [references/langsmith-mapping.md](references/langsmith-mapping.md) and apply it to the topology:

• One root run name per externally meaningful unit (chat-turn, job.summarise-thread, cron.nightly-research), run_type="chain".
• One child run name per role (supervisor, research-agent, writer-agent) and per tool (tool., run_type="tool"); model calls as llm., run_type="llm"; retrieval as retrieve., run_type="retriever".
• The thread key (session_id, or thread_id / conversation_id) on every run — set on the root and re-attached on each child, since metadata is not inherited.
• Tags: stack identifier, environment (dev / staging / prod), agent version, feature-flag arm if relevant.
• Metadata keys: a small consistent set so charts can group — agent_role, tool_id, model_id, prompt_version, user_tier, gateway_origin (when Ratel is present).

The plan should list every run name / tag / metadata key it introduces in one table the customer can paste into a shared doc, citing references/langsmith-mapping.md. The dashboard spec (Step 5) and /ratel-langsmith-analyze both read this table; if it's missing they can't function. The mapping renders the vendor-neutral conventions at [../ratel-observability-assessment/references/semantic-conventions.md](../ratel-observability-assessment/references/semantic-conventions.md) onto LangSmith primitives — do not re-derive the rules.

Step 4 — Spec the per-file instrumentation changes

For each file that needs wiring, cite the matching pattern from the stack reference (references/stack-*.md) rather than re-deriving it. Each entry: file path, what to wrap, which run type to use (chain / tool / llm / retriever), and what name / tags / metadata to attach per the Step 3 table — including the thread key, which must be present on every run.

Step 5 — Spec the dashboards (two groups)

Open [references/langsmith-value-map.md](references/langsmith-value-map.md) and [../ratel-observability-assessment/references/general-agent-dashboards.md](../ratel-observability-assessment/references/general-agent-dashboards.md). First, note what the built-in Monitor tab already provides per project (Traces, LLM Calls, Cost & Tokens, Tools, Run Types, Feedback Scores) — do not respec those as custom charts. Then pick the custom-chart subset that matches what's actually instrumented.

Default selection:

• Ratel-value group (only if Ratel is in or coming): Token Cost & Savings, Retrieval Quality, Gateway Origin Split, Upstream Health, Skill Retrieval Health (skills shipped in the v0.1.6 line). Add roadmap-conditional ones only if the customer is on a Ratel pre-release that has the feature, or has explicitly signed up to adopt it (e.g., Suggestion Adoption for v0.1.9). Names and chart specs come from [references/langsmith-value-map.md](references/langsmith-value-map.md).
• Agent-health group: Latency & Cost Overview, Error Surface, Tool Usage, Session Quality, Model & Prompt Drift. From [../ratel-observability-assessment/references/general-agent-dashboards.md](../ratel-observability-assessment/references/general-agent-dashboards.md) — but skip any that the Monitor tab already covers for this project, and render the rest as LangSmith custom charts (a Langfuse "observation" filter becomes a run type/run name filter; a Langfuse "trace" filter becomes is_root).

For each dashboard section, in this order:

1. Name — short, action-oriented (Token Cost & Savings, not Dashboard 1).
2. Why it matters — one paragraph, plain English; the customer's PM should know whether it's for them.
3. Required data — the run name(s), run type(s), tags, and metadata keys it depends on. Anything missing from the Step 3 table is a TODO blocker listed here.
4. Charts — for each, fill in the five fields from the chart vocabulary in [references/langsmith-value-map.md](references/langsmith-value-map.md): metric (y-axis); filter (run name / run type / tag / metadata / is_root); group-by (tag / metadata / run name / run type — top 5) or explicit Data Series; chart type (line / bar). Use Data Series for an explicit A/B split when grouping's top-5 limit isn't enough.
5. Pivots / drill-downs — what to click when a chart shows something odd, and where that drill-down lives (a saved run-table filter the customer reuses once the chart exists).
6. Roadmap footnote (Ratel dashboards only) — if a dashboard would benefit from a Ratel feature that hasn't shipped, name the feature and target version. Reuse the version map in [../ratel-observability-assessment/references/ratel-value-map.md](../ratel-observability-assessment/references/ratel-value-map.md) so this stays current.

Drop any dashboard whose backing data isn't instrumented. Skip rather than fake.

Step 6 — Decide Ratel-aware hooks (only if Ratel is or will be present)

Check whether @ratel-ai/sdk, the Python ratel-ai package, ratel-ai-core, or the ratel-mcp / @ratel-ai/mcp-server package appears anywhere in the manifest. If yes — or if the customer is signing up to add Ratel as part of this engagement — read [references/ratel-hooks.md](references/ratel-hooks.md) and add a section to the plan covering:

• Mapping each Ratel trace event onto a LangSmith run (search → run_type: tool named ratel.search_capabilities, skill_search → ratel.skill_search, get_skill_content → ratel.get_skill_content, InvokeStart/InvokeEnd → run_type: tool named ratel.invoke_tool, etc.). Names are reserved in [references/langsmith-value-map.md](references/langsmith-value-map.md).
• Required metadata on each run: gateway_origin (direct vs agent), top_k, hit_count, replace_mode, score of top hit, latency — plus the thread key, re-attached because metadata is not inherited.
• A "before / after" annotation strategy so the customer can run an A/B comparison once Ratel is wired in (the feature_flag tag split, charted via group-by or Data Series, is the A/B surface).

If Ratel is not present and there is no plan to introduce it, skip this section entirely. Do not pre-bake a Ratel sales pitch into a customer-owned doc — keep the plan honest.

Step 7 — Write the plan

Write the plan to /.ratel/ratel-langsmith-integrate.md (create the .ratel/ directory if it doesn't exist; ask the user to confirm the path if the repo already uses a different docs convention).

The plan must contain, in this order:

1. Summary — one paragraph: stack detected, agent topology, what's already instrumented (if anything), what this plan adds (wiring + dashboards), how many dashboards in each group.
2. Setup — SDK install commands (langsmith for Python / JS), env vars (LANGSMITH_TRACING=true, LANGSMITH_API_KEY, LANGSMITH_PROJECT, LANGSMITH_ENDPOINT), LangSmith MCP server registration steps (the langsmith-mcp-server config for the customer's Claude Code / Cursor, so /ratel-langsmith-analyze can read runs later), and a working "hello trace" snippet they can paste and verify.
3. Topology — the diagram from Step 2.
4. LangSmith naming/mapping table — the table from Step 3, citing [references/langsmith-mapping.md](references/langsmith-mapping.md).
5. Per-file instrumentation changes — from Step 4, citing the matching references/stack-*.md.
6. Dashboard build-spec — the two groups (Ratel-value + agent-health) from Step 5, noting what the built-in Monitor tab already covers, and citing [references/langsmith-value-map.md](references/langsmith-value-map.md) and [../ratel-observability-assessment/references/general-agent-dashboards.md](../ratel-observability-assessment/references/general-agent-dashboards.md).
7. Ratel hooks (conditional, per Step 6), citing [references/ratel-hooks.md](references/ratel-hooks.md).
8. Verification checklist — copy from [references/verification-checklist.md](references/verification-checklist.md): six items the customer can tick once instrumentation lands.
9. Out of scope (for now) — dashboards we'd add once

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: ratel-ai
• Source: ratel-ai/skills
• License: MIT

Install and usage instructions live in the source repository linked above.