# Genai Otel Instrument > GenAI OpenTelemetry Auto-Instrumentation Library A comprehensive wrapper for automatic instrumentation of LLM/GenAI applications Supports all major LLM providers and MCP (Model Context Protocol) tool calls - **Type:** MCP server - **Install:** `agentstack add mcp-mandark-droid-genai-otel-instrument` - **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior - **Seller:** [Mandark-droid](https://agentstack.voostack.com/s/mandark-droid) - **Installs:** 0 - **Category:** [AI & ML](https://agentstack.voostack.com/c/ai-and-ml) - **Latest version:** 0.1.0 - **License:** Apache-2.0 - **Upstream author:** [Mandark-droid](https://github.com/Mandark-droid) - **Source:** https://github.com/Mandark-droid/genai_otel_instrument ## Install ```sh agentstack add mcp-mandark-droid-genai-otel-instrument ``` Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent. ## About # TraceVerde **The most comprehensive OpenTelemetry auto-instrumentation library for LLM/GenAI applications** *Trace from OpenTelemetry traces. Verde meaning green - for sustainable, transparent AI observability.* [Documentation](https://mandark-droid.github.io/genai_otel_instrument/) | [Examples](https://github.com/Mandark-droid/genai_otel_instrument/tree/main/examples) | [Discord](https://discord.gg/6SVz6VKK) | [PyPI](https://pypi.org/project/genai-otel-instrument/) [](https://badge.fury.io/py/genai-otel-instrument) [](https://pypi.org/project/genai-otel-instrument/) [](https://opensource.org/licenses/Apache-2.0) [](https://pepy.tech/project/genai-otel-instrument) [](https://pepy.tech/project/genai-otel-instrument) [](https://github.com/Mandark-droid/genai_otel_instrument) [](https://opentelemetry.io/) [](https://github.com/Mandark-droid/genai_otel_instrument/actions) --- ## Get Started in 30 Seconds ```bash pip install genai-otel-instrument ``` ```python import genai_otel genai_otel.instrument() # Your existing code works unchanged - traces, metrics, and costs are captured automatically import openai client = openai.OpenAI() response = client.chat.completions.create(model="gpt-4o-mini", messages=[{"role": "user", "content": "Hello!"}]) ``` That's it. No wrappers, no decorators, no config files. Every LLM call, database query, and agent interaction is automatically traced with full cost breakdown. ## Why TraceVerde? | Feature | TraceVerde | OpenLIT | Traceloop/OpenLLMetry | Langfuse | |---------|-----------|---------|----------------------|----------| | Zero-code setup | Yes | Yes | Yes | SDK required | | LLM providers | 20+ | 25+ | 15+ | Via integrations | | Multi-agent frameworks | 8 (CrewAI, LangGraph, ADK, AutoGen, OpenAI Agents, Pydantic AI, etc.) | Limited | Limited | Limited | | Cost tracking | Automatic (1,050+ models) | Manual config | Manual config | Manual config | | GPU metrics (NVIDIA + AMD) | Yes | No | No | No | | MCP tool instrumentation | Yes (databases, caches, vector DBs, queues) | Limited | Limited | No | | Evaluation (PII, toxicity, bias, hallucination, prompt injection) | Built-in (6 detectors) | No | No | Separate service | | OpenTelemetry native | Yes | Yes | Yes | Partial | | License | Apache-2.0 | Apache-2.0 | Apache-2.0 | MIT | ## What Gets Instrumented? ### LLM Providers (20+) OpenAI, OpenRouter, CometAPI, Anthropic, Google AI, Google GenAI, AWS Bedrock, Azure OpenAI, Cohere, Mistral AI, Together AI, Groq, Ollama, Vertex AI, Replicate, HuggingFace, SambaNova, Sarvam AI, Hyperbolic, LiteLLM [See all providers with examples >>](https://mandark-droid.github.io/genai_otel_instrument/guides/llm-providers/) ### Multi-Agent Frameworks (8) CrewAI, LangGraph, Google ADK, AutoGen, AutoGen AgentChat, OpenAI Agents SDK, Pydantic AI, AWS Bedrock Agents [See all frameworks with examples >>](https://mandark-droid.github.io/genai_otel_instrument/guides/multi-agent-frameworks/) ### MCP Tools (20+) **Databases:** PostgreSQL, MySQL, MongoDB, SQLAlchemy, TimescaleDB, OpenSearch, Elasticsearch, FalkorDB **Caching:** Redis | **Queues:** Kafka, RabbitMQ | **Storage:** MinIO **Vector DBs:** Pinecone, Weaviate, Qdrant, ChromaDB, Milvus, FAISS, LanceDB [See all MCP tools >>](https://mandark-droid.github.io/genai_otel_instrument/guides/mcp-tools/) ### Built-in Evaluation (6 Detectors) PII Detection (GDPR/HIPAA/PCI-DSS), Toxicity Detection, Bias Detection, Prompt Injection Detection, Restricted Topics, Hallucination Detection [See all evaluation features with examples >>](https://mandark-droid.github.io/genai_otel_instrument/guides/evaluation/) ## Screenshots OpenAI traces with token usage, costs, and latency More screenshots ### Ollama (Local LLM) ### SmolAgents with Tool Calls ### GPU Metrics ### OpenSearch Dashboard ## Key Features ### Automatic Cost Tracking 1,050+ models across 30+ providers with per-request cost breakdown. Supports differential pricing (prompt vs completion), reasoning tokens, cache pricing, and custom model pricing. ```python # Cost tracking is enabled by default - just instrument and go genai_otel.instrument() # Or add custom pricing for proprietary models export GENAI_CUSTOM_PRICING_JSON='{"chat":{"my-model":{"promptPrice":0.001,"completionPrice":0.002}}}' ``` [Cost tracking guide >>](https://mandark-droid.github.io/genai_otel_instrument/guides/cost-tracking/) ### GPU Metrics (NVIDIA + AMD) Real-time monitoring of utilization, memory, temperature, power, PCIe throughput, throttling, and ECC errors. Multi-GPU aggregate metrics included. ```bash pip install genai-otel-instrument[gpu] # NVIDIA pip install genai-otel-instrument[amd-gpu] # AMD ``` [GPU metrics guide >>](https://mandark-droid.github.io/genai_otel_instrument/guides/gpu-metrics/) ### Multi-Agent Tracing Complete span hierarchy for agent frameworks with automatic context propagation: ``` Crew Execution +-- Agent: Senior Researcher (gpt-4) | +-- Task: Research OpenTelemetry | +-- openai.chat.completions (tokens: 1250, cost: $0.03) +-- Agent: Technical Writer (ollama:llama2) +-- Task: Write blog post +-- ollama.chat (tokens: 890, cost: $0.00) ``` ### Multimodal Observability (v1.1.0) First-class capture of image, audio, video, and document content parts on OpenAI, Anthropic, Google Gemini, and Groq spans. Bytes are offloaded to your configured object store (MinIO / S3 / filesystem / HTTP) and referenced from spans by URI — they never appear inline in span attributes. ```bash # Opt in (default is off — text-only behaviour is byte-identical to 1.0.x) export GENAI_OTEL_MEDIA_CAPTURE_MODE=full export GENAI_OTEL_MEDIA_STORE=minio export GENAI_OTEL_MEDIA_STORE_ENDPOINT=http://localhost:9000 export GENAI_OTEL_MEDIA_STORE_ACCESS_KEY=... export GENAI_OTEL_MEDIA_STORE_SECRET_KEY=... # Optional: plug in a redactor before upload export GENAI_OTEL_MEDIA_REDACTOR=genai_otel.media.redactors.face_blur ``` Spans get two co-emitted representations of the same multimodal content: - A **flat, queryable attribute namespace** — `gen_ai.prompt.{n}.content.{m}.{type, media_uri, media_mime_type, media_byte_size, media_source}` plus a `gen_ai.completion.*` mirror — for backends that index on flat attributes. - The **upstream-canonical `gen_ai.input.messages` / `gen_ai.output.messages` JSON** conforming to the [gen-ai message schemas](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-input-messages.json) in the dedicated `semantic-conventions-genai` repo, including the `document` modality, optional `byte_size`, and `stripped_reason` shape standardised by our upstream PRs #142 / #143 / #144 (see [Standards Contributions](#opentelemetry-standards-contributions) below). [Multimodal guide >>](https://mandark-droid.github.io/genai_otel_instrument/guides/multimodal/) ### Safety & Evaluation ```python genai_otel.instrument( enable_pii_detection=True, # GDPR/HIPAA/PCI-DSS compliance enable_toxicity_detection=True, # Perspective API + Detoxify enable_bias_detection=True, # 8 bias categories enable_prompt_injection_detection=True, enable_hallucination_detection=True, enable_restricted_topics=True, ) ``` [Evaluation guide with 50+ examples >>](https://mandark-droid.github.io/genai_otel_instrument/guides/evaluation/) ## Configuration ```bash # Required export OTEL_SERVICE_NAME=my-llm-app export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 # Optional export GENAI_ENABLE_GPU_METRICS=true export GENAI_ENABLE_COST_TRACKING=true export GENAI_SAMPLING_RATE=0.5 # Reduce volume in production export GENAI_ENABLED_INSTRUMENTORS=openai,crewai # Select specific instrumentors ``` [Full configuration reference >>](https://mandark-droid.github.io/genai_otel_instrument/getting-started/configuration/) ## Backend Integration Works with any OpenTelemetry-compatible backend: Jaeger, Zipkin, Prometheus, Grafana, Datadog, New Relic, Honeycomb, AWS X-Ray, Google Cloud Trace, Elastic APM, Splunk, SigNoz, self-hosted OTel Collector Pre-built [Grafana dashboard templates](https://github.com/Mandark-droid/genai_otel_instrument/tree/main/dashboards) included. ## Examples 90+ ready-to-run examples covering every provider, framework, and evaluation feature: ``` examples/ +-- openai/ # OpenAI chat, embeddings +-- anthropic/ # Anthropic + PII/toxicity detection +-- ollama/ # Local models + all evaluation features +-- crewai_example.py # Multi-agent crew orchestration +-- langgraph_example.py # Stateful graph workflows +-- google_adk_example.py # Google Agent Development Kit +-- autogen_example.py # Microsoft AutoGen agents +-- pii_detection/ # 10 PII examples (GDPR, HIPAA, PCI-DSS) +-- toxicity_detection/ # 8 toxicity examples +-- bias_detection/ # 8 bias examples (hiring compliance, etc.) +-- prompt_injection/ # 6 injection defense examples +-- hallucination/ # 4 hallucination detection examples +-- ... # And many more ``` [Browse all examples >>](https://github.com/Mandark-droid/genai_otel_instrument/tree/main/examples) ## OpenTelemetry Standards Contributions TraceVerde isn't only a consumer of OpenTelemetry GenAI semantic conventions — production gaps surfaced by the library are being upstreamed back into the spec. Active proposals on [`open-telemetry/semantic-conventions-genai`](https://github.com/open-telemetry/semantic-conventions-genai): | PR | Proposal | Status | |---|---|---| | [**#142**](https://github.com/open-telemetry/semantic-conventions-genai/pull/142) | Add `document` to the `Modality` enum on `BlobPart` / `FilePart` / `UriPart` — PDFs, DOCX, and other non-image/video/audio payloads currently fall through to the free-form `string` branch. BFSI KYC extraction is a high-volume real example. | Approved (@MikeGoldsmith) | | [**#143**](https://github.com/open-telemetry/semantic-conventions-genai/pull/143) | Add optional `byte_size` on the three media-part types so consumers get a uniform handle on payload size whether the content was carried inline, by URI, or by provider file id — useful for cost-of-capture telemetry and storage planning. Pydantic `ge=0` → JSON schema `"minimum": 0`. | Under review | | [**#144**](https://github.com/open-telemetry/semantic-conventions-genai/pull/144) | Make `content` / `file_id` / `uri` optional and add a free-form `stripped_reason` (`size_exceeded`, `modality_not_allowed`, `redactor_error`, `upload_error`, `no_store_configured`) so an instrumentation can fail-closed — record that it observed a media part but intentionally did not capture its bytes — while preserving the original part `type` and `modality`. Enforced via a top-level `anyOf` so structurally-empty parts cannot validate. | Under review | All three were migrated from the closed [`open-telemetry/semantic-conventions#3673`](https://github.com/open-telemetry/semantic-conventions/pull/3673) after the GenAI conventions split into the dedicated repo on 2026-05-05. Each PR ships under the new repo's V2 Weaver schema with the corresponding `models.ipynb` updates and `make check-policies` / `make generate-all` validation. TraceVerde v1.1.1 already emits the proposed shape on the wire via dual-emission (`OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai`), providing the reference implementation for these conventions. ## Ecosystem & Framework Contributions Beyond the spec, `genai_otel` is being upstreamed **into agent frameworks** as their native OpenTelemetry observability layer — demonstrating the library powering real third-party agents, not just first-party services: | PR | Framework | Contribution | Status | |---|---|---|---| | [**hermes-agent #48184**](https://github.com/NousResearch/hermes-agent/pull/48184) | [NousResearch Hermes](https://github.com/NousResearch/hermes-agent) | A bundled `observability/otel` plugin exporting Hermes turns / LLM calls / tool calls as OTel GenAI spans **and** dashboard log records, with no changes to Hermes core. When `genai-otel-instrument` is installed it additionally unlocks on-prem **GPU / energy / CO2 metrics**, **local-model cost** (parameter-size pricing for Ollama / HF / vLLM), and an inline **eval / guardrail suite** (PII, toxicity, bias, prompt-injection, restricted-topics, hallucination) scored on prompt **and** response — signals no vanilla OTel SDK or other GenAI instrumentor emits. | Open | ## Who Uses TraceVerde? TraceVerde is used by developers and teams building production GenAI applications. If you're using TraceVerde, we'd love to hear from you! [Add your company](https://github.com/Mandark-droid/genai_otel_instrument/issues/new?title=Add+my+company+to+users+list&labels=users) | [Join Discord](https://discord.gg/6SVz6VKK) ## Community - [Documentation](https://mandark-droid.github.io/genai_otel_instrument/) - Full guides, API reference, and tutorials - [Discord](https://discord.gg/6SVz6VKK) - Chat with the community - [GitHub Issues](https://github.com/Mandark-droid/genai_otel_instrument/issues) - Bug reports and feature requests - [Contributing](https://mandark-droid.github.io/genai_otel_instrument/community/contributing/) - How to contribute ## License Copyright 2025 Kshitij Thakkar. Licensed under the [Apache License 2.0](LICENSE). ## Source & license This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code. - **Author:** [Mandark-droid](https://github.com/Mandark-droid) - **Source:** [Mandark-droid/genai_otel_instrument](https://github.com/Mandark-droid/genai_otel_instrument) - **License:** Apache-2.0 Install and usage instructions live in the source repository linked above. ## Pricing - **Free** — Free ## Security capabilities Automated source analysis of v0.1.0 — what this tool can access: - **Network access:** no - **Filesystem access:** no - **Shell / process execution:** no - **Environment & secrets:** no - **Dynamic code execution:** no *"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.* ## Versions - **0.1.0** — security scan: passed — Imported from the upstream source. ## Links - Listing page: https://agentstack.voostack.com/l/mcp-mandark-droid-genai-otel-instrument - Seller: https://agentstack.voostack.com/s/mandark-droid - Browse the marketplace: https://agentstack.voostack.com/browse --- Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.