# Otel Collector Config

> >-

- **Type:** Skill
- **Install:** `agentstack add skill-s3onghyun-otelcol-doctor-otelcol-doctor`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [s3onghyun](https://agentstack.voostack.com/s/s3onghyun)
- **Installs:** 0
- **Category:** [Cloud & Infrastructure](https://agentstack.voostack.com/c/cloud-infrastructure)
- **Latest version:** 0.1.0
- **License:** Apache-2.0
- **Upstream author:** [s3onghyun](https://github.com/s3onghyun)
- **Source:** https://github.com/s3onghyun/otelcol-doctor

## Install

```sh
agentstack add skill-s3onghyun-otelcol-doctor-otelcol-doctor
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# OpenTelemetry Collector config — doctor

Generate correct configs from a plain-English description, and diagnose broken
ones. The Collector is a small set of concepts with a lot of sharp edges; this
skill encodes the edges so the output validates on the first try.

## Mental model (get this right and everything follows)

A Collector config has four component blocks plus a `service` block that wires
them:

```
receivers:   # where data comes IN  (otlp, prometheus, hostmetrics, filelog, kafka…)
processors:  # what happens in the MIDDLE, in order (memory_limiter, batch, transform…)
exporters:   # where data goes OUT  (otlp, otlphttp, debug, prometheus, prometheusremotewrite…)
connectors:  # bridge one pipeline's OUTPUT into another's INPUT (spanmetrics, routing…)
extensions:  # collector-level features, not in the data path (health_check, pprof, zpages…)

service:
  extensions: [health_check]
  pipelines:
    traces:  { receivers: [...], processors: [...], exporters: [...] }
    metrics: { receivers: [...], processors: [...], exporters: [...] }
    logs:    { receivers: [...], processors: [...], exporters: [...] }
  telemetry: { ... }   # the Collector's OWN logs/metrics
```

**The #1 rule everyone violates:** defining a component under `receivers:` /
`processors:` / `exporters:` does **nothing** on its own. A component only runs
if it is referenced inside a `service.pipelines.` list. A config can be
"valid" yet do nothing because a pipeline was never wired. Always check the
`service` block last, and confirm every component you defined is actually used
(and every name used is actually defined — that one fails validation).

**Pipelines are per-signal.** `traces`, `metrics`, `logs` are separate
pipelines. A receiver/exporter must support the signal of the pipeline it sits
in (e.g. the `prometheus` receiver is metrics-only; putting it in a `traces`
pipeline fails). You may run multiple named pipelines of the same signal
(`metrics/internal`, `metrics/app`).

## Workflow

### 1. Pin the three questions
- **Signals?** traces / metrics / logs (any subset).
- **Sources (receivers)?** app via OTLP? scrape Prometheus targets? host metrics? tail log files? Kafka?
- **Destinations (exporters)?** an OTLP backend (Tempo/Jaeger/vendor)? Prometheus (pull) or remote-write (push)? Loki? Mimir?

### 2. Choose the distribution — core vs contrib (this trips people constantly)
Many common components ship **only** in `otelcol-contrib`, not core `otelcol`:
`prometheus` receiver, `hostmetrics`, `filelog`, `kafka`, `resourcedetection`,
`transform`/`filter` (OTTL) processors, `tail_sampling`, the `spanmetrics`
connector, `prometheusremotewrite`/`loki` exporters. If the config uses any of
these, it needs the **contrib** image (`otel/opentelemetry-collector-contrib`)
or a custom build via the OpenTelemetry Collector Builder (`ocb`). Calling for a
contrib component on the core image is a startup crash, not a YAML error — flag it.

### 3. Order processors correctly (order is significant and load-bearing)
Processors run **in the listed order**. The canonical safe order:

```
processors: [memory_limiter, , , batch]
```

- **`memory_limiter` goes FIRST.** It must see data before anything buffers it,
  otherwise it can't shed load. Putting `batch` before it defeats the purpose.
- **`batch` goes LAST** (just before export) so it batches the final shape.
- **`tail_sampling`** (traces) must come before `batch`, and needs whole traces —
  don't shard the same trace across replicas without a load-balancing exporter
  in front.

### 4. Pick exporters deliberately
- `otlp` = OTLP over **gRPC** (default 4317). `otlphttp` = OTLP over **HTTP** (4318). Match the backend's port/protocol.
- `debug` is the console exporter (use `verbosity: detailed` while debugging). The old `logging` exporter is **removed/deprecated → use `debug`**.
- Metrics to Prometheus: **`prometheus`** = Collector exposes a `/metrics` endpoint for Prometheus to **scrape** (pull). **`prometheusremotewrite`** = Collector **pushes** to a remote-write endpoint (Mimir/Thanos/Cortex/Prometheus `--web.enable-remote-write-receiver`). Don't confuse pull vs push.
- Loki: the dedicated `loki` exporter was **removed** from contrib — send OTLP to Loki's native OTLP endpoint via `otlphttp` (`endpoint: http://loki:3100/otlp`).
- Tempo/Jaeger: send via `otlp`/`otlphttp` — the `jaeger` **exporter** is gone (Jaeger ingests OTLP natively). The `jaeger` **receiver** still exists (beta, in core+contrib) for *ingesting* legacy Jaeger-protocol spans during a migration; don't call it removed.

### 5. Always include the safety/ops basics
- `memory_limiter` processor in every pipeline that can be flooded.
- `health_check` extension (and wire it into `service.extensions`).
- For secrets/endpoints use env expansion: `endpoint: ${env:OTLP_ENDPOINT}` (note the `env:` prefix — bare `${VAR}` is deprecated syntax).
- Set `service.telemetry.logs.level` and, if needed, expose the Collector's own metrics.

### 6. Validate before declaring done
Never hand back a config you haven't validated. The Collector has a built-in
validator that loads and type-checks the full config without starting it:

```
otelcol validate --config=config.yaml          # core
otelcol-contrib validate --config=config.yaml   # contrib
```

The bundled `scripts/validate.sh` wraps this (auto-detects the contrib binary,
falls back to a Docker run if no local binary). If `otelcol` isn't available,
say so explicitly rather than claiming the config is validated.

## Fixing an existing config — run this checklist

When handed a broken or "it starts but nothing arrives" config, check in order:

1. **Unwired components** — every receiver/processor/exporter defined but not referenced in any `service.pipelines.*`? (silent no-op)
2. **Undefined references** — a name used in a pipeline that isn't defined above? (validation error)
3. **Signal mismatch** — a metrics-only receiver in a traces pipeline, etc.
4. **core vs contrib** — a contrib-only component on the core image? (crash on start)
5. **Processor order** — `batch` before `memory_limiter`? `memory_limiter` not first?
6. **pull vs push exporter** — `prometheus` where `prometheusremotewrite` was meant (or vice-versa)?
7. **Removed components** — `logging` exporter (→`debug`), `loki` exporter (→`otlphttp`), `jaeger` **exporter** (→`otlp`; the jaeger *receiver* is still fine), bare `${VAR}` env syntax (→`${env:VAR}`).
8. **Endpoint/protocol** — gRPC vs HTTP port mismatch (4317 vs 4318); `tls`/`insecure` set correctly for the environment.
9. **Endpoint binding** — receiver bound to `localhost` when traffic comes from other containers (needs `0.0.0.0`), or bound to `0.0.0.0` in a context where that's a security concern.

State the diagnosis as "what was wrong → why it failed → the fix", then output
the corrected config and validate it.

## The hard parts — get these right every time (see `references/advanced.md`)

A capable model already handles the basics above. These five are the ones it gets
*inconsistently* right; treat them as load-bearing and pull the exact patterns
from `references/advanced.md`:

1. **OTTL syntax** (`transform`/`filter` processors). It's a function language —
   `set(attributes["x"], "y") where `, `delete_key(...)`, `replace_pattern(...)`
   — **not** `attributes["x"] = "y"`, not jq, not SQL. Hallucinated OTTL is the #1
   advanced failure. Always set `error_mode`.
2. **spanmetrics (and any connector) is wired in TWO pipelines** — as an *exporter*
   in the source pipeline and a *receiver* in the destination pipeline. Wiring it
   once silently produces no metrics (or a validation error).
3. **`resource_to_telemetry_conversion: enabled: true`** on the Prometheus/remote-write
   exporter — without it, resource attributes (`service.name`, `deployment.environment`)
   land only on `target_info`, not as labels on each series. This is the usual
   "why is my label missing" cause.
4. **`tail_sampling` at >1 replica needs a `loadbalancing` tier** keyed by `traceID`,
   so all spans of a trace reach the same sampler instance. Tail-sampling directly
   behind a plain load balancer is broken sampling. `tail_sampling` goes before `batch`.
5. **Exporter reliability** — production exporters need `retry_on_failure` +
   `sending_queue` (and a `file_storage`-backed queue if data loss is unacceptable).
   Defaults drop data on a transient backend blip.

When the request touches any of these, encode the pattern explicitly rather than
trusting recall — that's the difference this skill is for.

## References
- `references/components.md` — curated catalog of the most-used receivers / processors / exporters / connectors with the gotcha for each, and the core-vs-contrib split.
- `references/advanced.md` — the hard parts in depth: OTTL patterns, spanmetrics dual-wiring, `resource_to_telemetry_conversion`, tail_sampling load-balancing topology, exporter reliability.
- `examples/` — a before/after pair (broken → corrected, validated) and `spanmetrics-config.yaml`, a validated advanced config (spanmetrics connector + OTTL + remote-write labels + retry/queue).

## Output discipline
- Emit **valid YAML only** inside config blocks — no `...` placeholders that won't parse.
- Comment the non-obvious lines (why `memory_limiter` is first, why this exporter).
- Prefer the smallest config that satisfies the request; don't bolt on components the user didn't ask for.
- If a component is contrib-only, say so and name the image/build needed.

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [s3onghyun](https://github.com/s3onghyun)
- **Source:** [s3onghyun/otelcol-doctor](https://github.com/s3onghyun/otelcol-doctor)
- **License:** Apache-2.0

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-s3onghyun-otelcol-doctor-otelcol-doctor
- Seller: https://agentstack.voostack.com/s/s3onghyun
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.