# Otel Genai Instrumentation

> >

- **Type:** Skill
- **Install:** `agentstack add skill-honeycombio-agent-skill-otel-genai-instrumentation`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [honeycombio](https://agentstack.voostack.com/s/honeycombio)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [honeycombio](https://github.com/honeycombio)
- **Source:** https://github.com/honeycombio/agent-skill/tree/main/honeycomb/skills/otel-genai-instrumentation

## Install

```sh
agentstack add skill-honeycombio-agent-skill-otel-genai-instrumentation
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# GenAI Instrumentation for Honeycomb

Instrumenting LLM and agent applications using OTel Semantic Conventions for GenAI
(currently v1.40.0, Development status). For conceptual foundations, see
the **observability-fundamentals** skill.

## Base OTEL Setup (Required First)

**BEFORE implementing GenAI instrumentation, ensure your base OpenTelemetry configuration is complete.**

Use the **otel-instrumentation** skill to configure all standard OTEL environment variables
(OTEL_SERVICE_NAME, OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS, OTEL_EXPORTER_OTLP_PROTOCOL,
signal-specific endpoints, etc.) and verify basic spans are flowing to Honeycomb.

GenAI instrumentation adds GenAI-specific configuration on top of that base setup.

## Critical Requirements (Non-Negotiable)

**BEFORE implementing any GenAI instrumentation, complete these steps in order:**

### Step 1: Ask About Content Capture (FIRST!)

**Stop and ask the user this question BEFORE writing any code or configuration:**

> "Do you want to capture the actual prompts and model responses in your traces?
>
> **Enabling content capture:**
> - ✅ Helps debug tool call failures, planning loops, and agent deadlocks
> - ✅ Lets you see why the model made specific decisions
> - ❌ Captures potentially sensitive content (user prompts, model responses)
> - ❌ May contain PII, proprietary data, or confidential information
>
> **Recommended for:** debugging/development, non-sensitive data, or if you have filtering
>
> **Not recommended for:** production with sensitive data, PII/health/financial info"

**Record their answer** — you'll need it when configuring instrumentation.

### Step 2: Enable GenAI Conventions (REQUIRED)

```bash
export OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental
```

Without this, GenAI spans will not be created.

### Step 3: Set Required Attributes on EVERY Span (REQUIRED)

- `gen_ai.operation.name` — e.g., `chat`, `execute_tool`, `invoke_agent`
- `gen_ai.conversation.id` — same value for all spans in a conversation

**Impact if missing**: Spans won't be recognized as GenAI operations and cannot be queried by session.

### Step 4: Implement force_flush() (REQUIRED)

GenAI apps often exit early (crash, Ctrl+C, CLI). Force flush after each top-level invocation
to prevent silent span loss.

For OTLP configuration, environment variables, and Honeycomb authentication (including the
silent-rejection pitfall), see the **otel-instrumentation** skill.

## Prerequisites

**This skill assumes your agent application is already sending telemetry to Honeycomb.** You should have:
- OpenTelemetry SDK installed and initialized
- All standard OTEL environment variables configured (see **Base OTEL Setup** section above)
- OTLP exporter configured with your Honeycomb API key
- Basic spans flowing to Honeycomb

**If you haven't set this up yet, use the otel-instrumentation skill first** for:
- SDK setup and dependencies
- OTEL environment variables (OTEL_SERVICE_NAME, OTEL_EXPORTER_OTLP_*, etc.)
- OTLP configuration and Honeycomb authentication
- Verification that spans are flowing

Once base telemetry is working, return here to add GenAI-specific instrumentation.

## Auto-Instrumentation (Python and Node.js)

Python and Node.js have official OTel auto-instrumentation packages for GenAI providers.
Go, Java, etc. require manual instrumentation (section below).

### Python

| Package | Provider | Min SDK Version |
| :--- | :--- | :--- |
| `opentelemetry-instrumentation-openai-v2` | OpenAI | openai >= v1.26.0 |
| `opentelemetry-instrumentation-anthropic` | Anthropic | anthropic >= v0.16.0 |
| `opentelemetry-instrumentation-claude-agent-sdk` | Claude Agent SDK | claude-agent-sdk >= v0.1.14 |
| `opentelemetry-instrumentation-google-genai` | Google GenAI | google-genai >= v1.32.0 |
| `opentelemetry-instrumentation-vertexai` | Vertex AI | google-cloud-aiplatform >= v1.64 |
| `opentelemetry-instrumentation-langchain` | LangChain | langchain >= v0.3.21 |
| `opentelemetry-instrumentation-openai-agents-v2` | OpenAI Agents | openai-agents >= v0.3.3 |
| `opentelemetry-instrumentation-weaviate` | Weaviate | weaviate-client >= v3.0.0, ` + `Instrumentor().instrument()` or CLI
`opentelemetry-instrument`.

### Node.js

| Package | Provider | Min SDK Version |
| :--- | :--- | :--- |
| `@opentelemetry/instrumentation-openai` | OpenAI | openai >= 4.19.0 |
| `@opentelemetry/instrumentation-langchain` | LangChain | langchain >= 1.0.0 (not yet published to npm) |

Setup: `npm install ` + register via OTel Node SDK.

For per-provider install commands, upstream README links, and supported version
details, see
`${CLAUDE_PLUGIN_ROOT}/skills/otel-genai-instrumentation/references/auto-instrumentation-setup.md`.

## Manual Instrumentation

For languages without auto-instrumentation (Go, Java, etc.) or when
auto-instrumentation doesn't cover your needs.

Key patterns:
- Creating inference spans (`chat`, `text_completion`, `generate_content`)
- Creating embedding and retrieval spans
- Setting request attributes before the call, response/usage attributes after
- Error handling with `error.type` and span status

For code examples in Python, Node.js, and Go, see
`${CLAUDE_PLUGIN_ROOT}/skills/otel-genai-instrumentation/references/manual-instrumentation.md`.

## Span Flushing for GenAI Apps

**Critical for GenAI applications.** The `BatchSpanProcessor` buffers spans (default
5 s schedule delay). GenAI agent runs are long-lived but may exit before the batch
flushes — crash, Ctrl+C, short CLI invocations — causing **silent span loss**.

**Rule: force-flush after every top-level agent invocation.** Expose the span
processor and call `forceFlush()` without tearing down the SDK, so subsequent
invocations continue producing spans.

### Why `shutdown()` is wrong here

`sdk.shutdown()` tears down the entire pipeline — after shutdown, no new spans are
recorded. For apps that run multiple agent invocations (polling loops, HTTP servers,
CLI batch modes), you need spans to keep flowing. Use `forceFlush()` instead.

### Python

```python
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

span_processor = BatchSpanProcessor(exporter)
provider = TracerProvider()
provider.add_span_processor(span_processor)

async def flush_telemetry():
    """Flush pending spans without shutting down."""
    span_processor.force_flush()
```

### Node.js

```typescript
import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base";

let spanProcessor: BatchSpanProcessor | null = null;

export function initTelemetry(): void {
  // ... exporter setup ...
  spanProcessor = new BatchSpanProcessor(traceExporter);
  sdk = new NodeSDK({ spanProcessors: [spanProcessor], /* ... */ });
  sdk.start();
}

export async function flushTelemetry(): Promise {
  if (spanProcessor) {
    await spanProcessor.forceFlush();
  }
}
```

### Go

```go
var spanProcessor *sdktrace.BatchSpanProcessor

func InitTelemetry() {
    spanProcessor = sdktrace.NewBatchSpanProcessor(exporter)
    // ... provider setup ...
}

func FlushTelemetry(ctx context.Context) error {
    return spanProcessor.ForceFlush(ctx)
}
```

### Where to call `flushTelemetry()`

- **After each agent invocation** — ensures the full trace (agent + chat + tool spans)
  is exported before moving to the next task
- **In polling/server loops** — flush after processing each request or ticket
- **Before `process.exit()`** — as a safety net alongside `shutdownTelemetry()`
- **NOT inside the agent loop** — flushing per-chat-turn adds latency; flush once at
  the outer boundary

Example integration:
```typescript
for (const ticket of tickets) {
  await triageIssue(ticket);   // produces invoke_agent + chat + tool spans
  await flushTelemetry();      // ensure spans are exported before next ticket
}
```

For complete code examples showing flush integration with tool-calling loops, see
`${CLAUDE_PLUGIN_ROOT}/skills/otel-genai-instrumentation/references/manual-instrumentation.md`.

## GenAI Span Types

**Span names MUST follow the pattern `"{operation} {identifier}"`.** The `gen_ai.operation.name`
attribute and the span name prefix must match. For example, a span with
`gen_ai.operation.name = "invoke_agent"` must be named `"invoke_agent {agent_name}"`,
not `"mypackage.DoSomething"`.

| Operation | `gen_ai.operation.name` | SpanKind | Span Name |
| :--- | :--- | :--- | :--- |
| Chat/completion | `chat` | CLIENT | `chat {model}` |
| Text completion | `text_completion` | CLIENT | `text_completion {model}` |
| Content generation | `generate_content` | CLIENT | `generate_content {model}` |
| Embeddings | `embeddings` | CLIENT | `embeddings {model}` |
| RAG retrieval | `retrieval` | CLIENT | `retrieval {data_source}` |
| Tool execution | `execute_tool` | INTERNAL | `execute_tool {tool_name}` |
| Agent creation | `create_agent` | CLIENT | `create_agent {agent_name}` |
| Agent invocation | `invoke_agent` | CLIENT/INTERNAL | `invoke_agent {agent_name}` |
| Workflow step | `invoke_workflow` | INTERNAL | `invoke_workflow {workflow_name}` |

### Required Attributes on All GenAI Spans

**CRITICAL: Every GenAI span MUST include these two attributes. This is non-negotiable.**

1. **`gen_ai.operation.name`** — Identifies the operation type (`chat`, `embeddings`, `execute_tool`, `invoke_agent`, etc.).
   - **Without this**: The span is not recognized as a GenAI operation and will be excluded from GenAI-specific queries and visualizations in Honeycomb
   - **Set on EVERY span**: chat, execute_tool, invoke_agent, embeddings, retrieval, etc.

2. **`gen_ai.conversation.id`** — Ties operations together within a conversation or session.
   - **Without this**: Spans cannot be queried as part of a multi-operation workflow, breaking session-level analysis
   - **Use the SAME value** across all operations in a conversation thread (user request → agent invocation → chat calls → tool executions → responses)
   - Generate once at the start of a conversation, propagate to all operations

**When to set:** When creating the span (in the span attributes), not after.

**How to propagate conversation_id:**
- In-process: Pass as parameter or store in context
- HTTP/A2A: Include in request payload or propagate via headers

**Impact of missing these attributes:**
- Missing `gen_ai.operation.name` → Span not recognized as GenAI operation, excluded from GenAI-specific queries and visualizations
- Missing `gen_ai.conversation.id` → Span excluded from session queries, cannot correlate operations within a conversation, breaks multi-turn analysis

**What is a conversation?**

A conversation is a **customer session or user interaction**, NOT a single LLM call. One conversation contains:
- Multiple user turns/messages
- All LLM calls handling those turns
- All tool executions triggered by those LLM calls
- All agent invocations within that session

See the [OTel GenAI spec](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-spans/#conversation-id) for the definition. Key principle: use the same conversation.id when conversation history/context is maintained across operations.

**When to use the same conversation_id:**
- All operations within a single customer session
- All turns in a multi-turn interaction
- All LLM calls handling those turns
- All tool executions and agent invocations within that session
- Multiple agents participating in the same session

**Example:** User starts a support session. Over the next 10 minutes they send 5 messages. The assistant makes 15 LLM calls and executes 8 tools to handle those messages. ALL of these spans share the SAME conversation.id because they're part of one customer session.

**Common mistake:** Generating a new conversation_id for each LLM call. This breaks session-level analysis. Generate conversation_id ONCE at session start, reuse for all operations until session ends.

For trace structures showing how these spans compose (tool-calling loops, multi-turn
conversations, nested agents, workflows), see
`${CLAUDE_PLUGIN_ROOT}/skills/otel-genai-instrumentation/references/agent-and-tool-patterns.md`.

**A2A / HTTP-based agent delegation:** When agents communicate over HTTP (A2A protocol,
REST delegation), manually propagate both trace context (via headers) AND conversation.id
(via payload). Client: `propagation.inject()` + include conversation.id in request body.
Server: `propagation.extract()` + `context.with()` + extract conversation.id from payload
and pass to all operations. See the "A2A (Agent-to-Agent) HTTP Context Propagation"
section in the reference file above.

## Generating and Propagating Conversation ID

Generate conversation_id at your application's **session boundary**:
- Chat apps: when user opens new chat/thread
- Support systems: when customer starts session
- CLI tools: at command invocation
- HTTP APIs: when session/conversation is created
- Bots: when user starts thread/DM

Pass the SAME conversation_id to all operations within that session — all user turns, all LLM calls handling those turns, all tool executions, all agent invocations.

**Propagation methods:**
- In-process: store in session object, pass as parameter
- HTTP/microservices: include in request payload or header (`X-Conversation-ID`)
- Bots: store in state (Redis, DB), retrieve using thread/DM ID

## Attribute Completeness

**Set all attributes for which you have data available.** The OTel GenAI semantic conventions define comprehensive attributes for each operation type — if your application has the data (model name, tokens, tool arguments, etc.), set the corresponding attribute.

**Critical principle**: Don't selectively omit attributes. Incomplete instrumentation limits your ability to:
- Identify which models and agents were involved in a trace
- Track token usage and costs across operations
- Debug tool call failures (missing arguments/results)
- Understand conversation flow (missing messages)
- Correlate agent behavior with configuration (missing request parameters)

For the full attribute definitions by operation type, see the upstream semantic conventions:
- Model operations (chat, embeddings): https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-spans/
- Agent operations (invoke_agent, execute_tool): https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-agent-spans/
- Local reference: `${CLAUDE_PLUGIN_ROOT}/skills/otel-genai-instrumentation/references/genai-attributes-catalog.md`

**What "data available" means**:
- API response fields → set corresponding response attributes (model, tokens, finish_reasons, response_id)
- Request parameters → set request attributes (temperature, max_tokens, top_p, etc.)
- Agent metadata → set agent attributes (name, id, description, version)
- Tool execution → set tool attributes (name, call_id, arguments, result)
- Conversation context → set conversation_id on ALL GenAI spans (required, not optional) — use the same ID across all operations in a conversation thread

The code examples in this skill show core attributes for each operation type. For complete coverage, consult the upstream spec and instrument every attribute your application can populate.

**Impact of incomplete instrumentation**:

- Missing `gen_ai.operation.name` → span not recognized as GenAI operation, excluded from GenAI queries
- Missing `gen_ai.conversation.id` → span excluded from session queries, cannot correlate operations within a conversation
- Missing `gen_ai.request.model` / `gen_ai.response.model` → can't identify which model was used
- Missing `gen_ai.usage.*` tokens → can't track costs or identify expensive operations
- Missing `gen_ai.tool.call.arguments` / `gen_ai.tool.call.result` → can't debug why tools failed or returned unexpected results
- Missing `gen_ai.input.messages` / `gen_ai.output.messages` → can't see what prompted a response, can't debug planning loops or hallucinations
- Missing agent attributes → can't distinguish between agents in multi-agent systems
- Missing request para

…

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [honeycombio](https://github.com/honeycombio)
- **Source:** [honeycombio/agent-skill](https://github.com/honeycombio/agent-skill)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** yes
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-honeycombio-agent-skill-otel-genai-instrumentation
- Seller: https://agentstack.voostack.com/s/honeycombio
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.