Otel Genai Instrumentation

GenAI Instrumentation for Honeycomb

Instrumenting LLM and agent applications using OTel Semantic Conventions for GenAI (currently v1.40.0, Development status). For conceptual foundations, see the observability-fundamentals skill.

Base OTEL Setup (Required First)

BEFORE implementing GenAI instrumentation, ensure your base OpenTelemetry configuration is complete.

Use the otel-instrumentation skill to configure all standard OTEL environment variables (OTELSERVICENAME, OTELEXPORTEROTLPENDPOINT, OTELEXPORTEROTLPHEADERS, OTELEXPORTEROTLP_PROTOCOL, signal-specific endpoints, etc.) and verify basic spans are flowing to Honeycomb.

GenAI instrumentation adds GenAI-specific configuration on top of that base setup.

Critical Requirements (Non-Negotiable)

BEFORE implementing any GenAI instrumentation, complete these steps in order:

Step 1: Ask About Content Capture (FIRST!)

Stop and ask the user this question BEFORE writing any code or configuration:

> "Do you want to capture the actual prompts and model responses in your traces? > > Enabling content capture: > - ✅ Helps debug tool call failures, planning loops, and agent deadlocks > - ✅ Lets you see why the model made specific decisions > - ❌ Captures potentially sensitive content (user prompts, model responses) > - ❌ May contain PII, proprietary data, or confidential information > > Recommended for: debugging/development, non-sensitive data, or if you have filtering > > Not recommended for: production with sensitive data, PII/health/financial info"

Record their answer — you'll need it when configuring instrumentation.

Step 2: Enable GenAI Conventions (REQUIRED)

export OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental

Without this, GenAI spans will not be created.

Step 3: Set Required Attributes on EVERY Span (REQUIRED)

• gen_ai.operation.name — e.g., chat, execute_tool, invoke_agent
• gen_ai.conversation.id — same value for all spans in a conversation

Impact if missing: Spans won't be recognized as GenAI operations and cannot be queried by session.

Step 4: Implement force_flush() (REQUIRED)

GenAI apps often exit early (crash, Ctrl+C, CLI). Force flush after each top-level invocation to prevent silent span loss.

For OTLP configuration, environment variables, and Honeycomb authentication (including the silent-rejection pitfall), see the otel-instrumentation skill.

Prerequisites

This skill assumes your agent application is already sending telemetry to Honeycomb. You should have:

• OpenTelemetry SDK installed and initialized
• All standard OTEL environment variables configured (see Base OTEL Setup section above)
• OTLP exporter configured with your Honeycomb API key
• Basic spans flowing to Honeycomb

If you haven't set this up yet, use the otel-instrumentation skill first for:

• SDK setup and dependencies
• OTEL environment variables (OTELSERVICENAME, OTELEXPORTEROTLP_*, etc.)
• OTLP configuration and Honeycomb authentication
• Verification that spans are flowing

Once base telemetry is working, return here to add GenAI-specific instrumentation.

Auto-Instrumentation (Python and Node.js)

Python and Node.js have official OTel auto-instrumentation packages for GenAI providers. Go, Java, etc. require manual instrumentation (section below).

Python

| Package | Provider | Min SDK Version | | :--- | :--- | :--- | | opentelemetry-instrumentation-openai-v2 | OpenAI | openai >= v1.26.0 | | opentelemetry-instrumentation-anthropic | Anthropic | anthropic >= v0.16.0 | | opentelemetry-instrumentation-claude-agent-sdk | Claude Agent SDK | claude-agent-sdk >= v0.1.14 | | opentelemetry-instrumentation-google-genai | Google GenAI | google-genai >= v1.32.0 | | opentelemetry-instrumentation-vertexai | Vertex AI | google-cloud-aiplatform >= v1.64 | | opentelemetry-instrumentation-langchain | LangChain | langchain >= v0.3.21 | | opentelemetry-instrumentation-openai-agents-v2 | OpenAI Agents | openai-agents >= v0.3.3 | | opentelemetry-instrumentation-weaviate | Weaviate | weaviate-client >= v3.0.0, + Instrumentor().instrument() or CLI opentelemetry-instrument`.

Node.js

| Package | Provider | Min SDK Version | | :--- | :--- | :--- | | @opentelemetry/instrumentation-openai | OpenAI | openai >= 4.19.0 | | @opentelemetry/instrumentation-langchain | LangChain | langchain >= 1.0.0 (not yet published to npm) |

Setup: npm install + register via OTel Node SDK.

For per-provider install commands, upstream README links, and supported version details, see ${CLAUDE_PLUGIN_ROOT}/skills/otel-genai-instrumentation/references/auto-instrumentation-setup.md.

Manual Instrumentation

For languages without auto-instrumentation (Go, Java, etc.) or when auto-instrumentation doesn't cover your needs.

Key patterns:

• Creating inference spans (chat, text_completion, generate_content)
• Creating embedding and retrieval spans
• Setting request attributes before the call, response/usage attributes after
• Error handling with error.type and span status

For code examples in Python, Node.js, and Go, see ${CLAUDE_PLUGIN_ROOT}/skills/otel-genai-instrumentation/references/manual-instrumentation.md.

Span Flushing for GenAI Apps

Critical for GenAI applications. The BatchSpanProcessor buffers spans (default 5 s schedule delay). GenAI agent runs are long-lived but may exit before the batch flushes — crash, Ctrl+C, short CLI invocations — causing silent span loss.

Rule: force-flush after every top-level agent invocation. Expose the span processor and call forceFlush() without tearing down the SDK, so subsequent invocations continue producing spans.

Why shutdown() is wrong here

sdk.shutdown() tears down the entire pipeline — after shutdown, no new spans are recorded. For apps that run multiple agent invocations (polling loops, HTTP servers, CLI batch modes), you need spans to keep flowing. Use forceFlush() instead.

Python

from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

span_processor = BatchSpanProcessor(exporter)
provider = TracerProvider()
provider.add_span_processor(span_processor)

async def flush_telemetry():
    """Flush pending spans without shutting down."""
    span_processor.force_flush()

Node.js

import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base";

let spanProcessor: BatchSpanProcessor | null = null;

export function initTelemetry(): void {
  // ... exporter setup ...
  spanProcessor = new BatchSpanProcessor(traceExporter);
  sdk = new NodeSDK({ spanProcessors: [spanProcessor], /* ... */ });
  sdk.start();
}

export async function flushTelemetry(): Promise {
  if (spanProcessor) {
    await spanProcessor.forceFlush();
  }
}

Go

var spanProcessor *sdktrace.BatchSpanProcessor

func InitTelemetry() {
    spanProcessor = sdktrace.NewBatchSpanProcessor(exporter)
    // ... provider setup ...
}

func FlushTelemetry(ctx context.Context) error {
    return spanProcessor.ForceFlush(ctx)
}

Where to call flushTelemetry()

• After each agent invocation — ensures the full trace (agent + chat + tool spans)

is exported before moving to the next task

• In polling/server loops — flush after processing each request or ticket
• Before process.exit() — as a safety net alongside shutdownTelemetry()
• NOT inside the agent loop — flushing per-chat-turn adds latency; flush once at

the outer boundary

Example integration:

for (const ticket of tickets) {
  await triageIssue(ticket);   // produces invoke_agent + chat + tool spans
  await flushTelemetry();      // ensure spans are exported before next ticket
}

For complete code examples showing flush integration with tool-calling loops, see ${CLAUDE_PLUGIN_ROOT}/skills/otel-genai-instrumentation/references/manual-instrumentation.md.

GenAI Span Types

Span names MUST follow the pattern "{operation} {identifier}". The gen_ai.operation.name attribute and the span name prefix must match. For example, a span with gen_ai.operation.name = "invoke_agent" must be named "invoke_agent {agent_name}", not "mypackage.DoSomething".

| Operation | gen_ai.operation.name | SpanKind | Span Name | | :--- | :--- | :--- | :--- | | Chat/completion | chat | CLIENT | chat {model} | | Text completion | text_completion | CLIENT | text_completion {model} | | Content generation | generate_content | CLIENT | generate_content {model} | | Embeddings | embeddings | CLIENT | embeddings {model} | | RAG retrieval | retrieval | CLIENT | retrieval {data_source} | | Tool execution | execute_tool | INTERNAL | execute_tool {tool_name} | | Agent creation | create_agent | CLIENT | create_agent {agent_name} | | Agent invocation | invoke_agent | CLIENT/INTERNAL | invoke_agent {agent_name} | | Workflow step | invoke_workflow | INTERNAL | invoke_workflow {workflow_name} |

Required Attributes on All GenAI Spans

CRITICAL: Every GenAI span MUST include these two attributes. This is non-negotiable.

1. gen_ai.operation.name — Identifies the operation type (chat, embeddings, execute_tool, invoke_agent, etc.).

• Without this: The span is not recognized as a GenAI operation and will be excluded from GenAI-specific queries and visualizations in Honeycomb
• Set on EVERY span: chat, executetool, invokeagent, embeddings, retrieval, etc.

1. gen_ai.conversation.id — Ties operations together within a conversation or session.

• Without this: Spans cannot be queried as part of a multi-operation workflow, breaking session-level analysis
• Use the SAME value across all operations in a conversation thread (user request → agent invocation → chat calls → tool executions → responses)
• Generate once at the start of a conversation, propagate to all operations

When to set: When creating the span (in the span attributes), not after.

How to propagate conversation_id:

• In-process: Pass as parameter or store in context
• HTTP/A2A: Include in request payload or propagate via headers

Impact of missing these attributes:

• Missing gen_ai.operation.name → Span not recognized as GenAI operation, excluded from GenAI-specific queries and visualizations
• Missing gen_ai.conversation.id → Span excluded from session queries, cannot correlate operations within a conversation, breaks multi-turn analysis

What is a conversation?

A conversation is a customer session or user interaction, NOT a single LLM call. One conversation contains:

• Multiple user turns/messages
• All LLM calls handling those turns
• All tool executions triggered by those LLM calls
• All agent invocations within that session

See the OTel GenAI spec for the definition. Key principle: use the same conversation.id when conversation history/context is maintained across operations.

When to use the same conversation_id:

• All operations within a single customer session
• All turns in a multi-turn interaction
• All LLM calls handling those turns
• All tool executions and agent invocations within that session
• Multiple agents participating in the same session

Example: User starts a support session. Over the next 10 minutes they send 5 messages. The assistant makes 15 LLM calls and executes 8 tools to handle those messages. ALL of these spans share the SAME conversation.id because they're part of one customer session.

Common mistake: Generating a new conversationid for each LLM call. This breaks session-level analysis. Generate conversationid ONCE at session start, reuse for all operations until session ends.

For trace structures showing how these spans compose (tool-calling loops, multi-turn conversations, nested agents, workflows), see ${CLAUDE_PLUGIN_ROOT}/skills/otel-genai-instrumentation/references/agent-and-tool-patterns.md.

A2A / HTTP-based agent delegation: When agents communicate over HTTP (A2A protocol, REST delegation), manually propagate both trace context (via headers) AND conversation.id (via payload). Client: propagation.inject() + include conversation.id in request body. Server: propagation.extract() + context.with() + extract conversation.id from payload and pass to all operations. See the "A2A (Agent-to-Agent) HTTP Context Propagation" section in the reference file above.

Generating and Propagating Conversation ID

Generate conversation_id at your application's session boundary:

• Chat apps: when user opens new chat/thread
• Support systems: when customer starts session
• CLI tools: at command invocation
• HTTP APIs: when session/conversation is created
• Bots: when user starts thread/DM

Pass the SAME conversation_id to all operations within that session — all user turns, all LLM calls handling those turns, all tool executions, all agent invocations.

Propagation methods:

• In-process: store in session object, pass as parameter
• HTTP/microservices: include in request payload or header (X-Conversation-ID)
• Bots: store in state (Redis, DB), retrieve using thread/DM ID

Attribute Completeness

Set all attributes for which you have data available. The OTel GenAI semantic conventions define comprehensive attributes for each operation type — if your application has the data (model name, tokens, tool arguments, etc.), set the corresponding attribute.

Critical principle: Don't selectively omit attributes. Incomplete instrumentation limits your ability to:

• Identify which models and agents were involved in a trace
• Track token usage and costs across operations
• Debug tool call failures (missing arguments/results)
• Understand conversation flow (missing messages)
• Correlate agent behavior with configuration (missing request parameters)

For the full attribute definitions by operation type, see the upstream semantic conventions:

• Model operations (chat, embeddings): https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-spans/
• Agent operations (invokeagent, executetool): https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-agent-spans/
• Local reference: ${CLAUDE_PLUGIN_ROOT}/skills/otel-genai-instrumentation/references/genai-attributes-catalog.md

What "data available" means:

• API response fields → set corresponding response attributes (model, tokens, finishreasons, responseid)
• Request parameters → set request attributes (temperature, maxtokens, topp, etc.)
• Agent metadata → set agent attributes (name, id, description, version)
• Tool execution → set tool attributes (name, call_id, arguments, result)
• Conversation context → set conversation_id on ALL GenAI spans (required, not optional) — use the same ID across all operations in a conversation thread

The code examples in this skill show core attributes for each operation type. For complete coverage, consult the upstream spec and instrument every attribute your application can populate.

Impact of incomplete instrumentation:

• Missing gen_ai.operation.name → span not recognized as GenAI operation, excluded from GenAI queries
• Missing gen_ai.conversation.id → span excluded from session queries, cannot correlate operations within a conversation
• Missing gen_ai.request.model / gen_ai.response.model → can't identify which model was used
• Missing gen_ai.usage.* tokens → can't track costs or identify expensive operations
• Missing gen_ai.tool.call.arguments / gen_ai.tool.call.result → can't debug why tools failed or returned unexpected results
• Missing gen_ai.input.messages / gen_ai.output.messages → can't see what prompted a response, can't debug planning loops or hallucinations
• Missing agent attributes → can't distinguish between agents in multi-agent systems
• Missing request para

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: honeycombio
• Source: honeycombio/agent-skill
• License: MIT

Install and usage instructions live in the source repository linked above.