Signoz Setting Up Observability

Setting Up Observability After Ingestion

Take a service from "telemetry is flowing into SigNoz" to "I have a dashboard, alerts, saved views, and a tuning loop." This skill is an orchestration layer, not a standalone reference — the mechanical work lives in the sibling SigNoz skills and the MCP tools/resources hold the current payload schemas. This skill sequences them and supplies the judgment calls (scope, SLOs, thresholds, what to skip) that no single skill owns; it deliberately does not restate payload schemas or field rules, so read the relevant signoz://… resource before composing any payload.

Use this when traces, logs, or metrics are already landing in SigNoz and you want an opinionated, SLO-aware setup — not when you're still wiring up the SDK. Audience: two consumers — an autonomous AI SRE agent that runs without a human in the loop (e.g. an in-product SigNoz assistant), and an engineer at a Claude Code / Codex / Cursor prompt. Both follow the same flow; where a step needs a human decision (Phase 1 triage, Phase 5 sign-off), the host decides how to surface it, and an autonomous host fills the gap from upstream context instead of blocking.

Prerequisites

This skill calls SigNoz MCP server tools (signoz_list_services, signoz_list_dashboards, signoz_get_field_keys, signoz_execute_builder_query, signoz_create_dashboard, signoz_create_alert, signoz_create_view, etc.) and delegates to sibling skills. Before running the workflow, confirm the signoz_* tools are available. If they are not, the SigNoz MCP server is not installed or configured — run signoz-mcp-setup first. Do not fall back to raw HTTP calls or fabricate payloads.

When to use

Use this skill when the user wants the whole post-ingestion setup: SLI/SLO → exploration → dashboard → views → alerts → tuning, in any combination of those deliverables, sequenced as one workflow.

Do NOT use this skill when the user wants a single artifact — hand off directly:

• Just a dashboard → signoz-creating-dashboards.
• Just one static / threshold alert or notification rule →

signoz-creating-alerts. (But SLO / burn-rate / error-budget alerting — even with no dashboard — stays here: it needs the SLI/SLO capture and burn-rate judgment in the phases below.)

• Just a saved Explorer view → signoz-managing-views.
• A one-off exploratory query → signoz-generating-queries.
• A concept/doc lookup (SRE methods, OTel conventions, burn-rate math)

→ signoz-searching-docs / signoz_search_docs.

Each phase below points to the skill that does the mechanical work; this skill owns the order and the decisions.

Phase 1 — Triage scope (don't skip)

Before touching any tools, get these answers. Each unblocks a downstream phase.

1. What does this service do, and does it call external services?

One or two sentences. This shapes the SLI (what "working" means) and flags downstream dependencies worth monitoring — if it calls a DB, cache, queue, or third-party API, plan to watch the client-span latency and error rate to each dependency, since a healthy service fronting a sick dependency still fails users.

1. What deliverables? dashboard / alerts / saved views — any

combination.

1. Infra monitoring too? (optional — ask) If the service runs on

infra you own (k8s pods, VMs, containers) shipping telemetry under the same resource attributes, you can add USE/infra panels (CPU, memory, restarts, network) with no extra instrumentation — see Phase

1. Skip if a platform team already owns the host/cluster dashboard.
2. Which signals are emitted? traces / logs / metrics. Drives which

signoz_get_field_keys calls you make.

1. What's the canonical resource filter? Usually service.name.

Confirm the exact value before querying — don't guess.

1. What does success look like for users of this service? (Sets up

Phase 2.) Even a one-line answer ("requests under 1s with no 5xx") is enough — it becomes the SLI.

1. Who owns it, and is there a runbook? Owning team, service tier,

and the prod/staging scope — these become alert labels and annotations in Phase 8. Capture a real runbook URL if one exists; never invent one.

Offer "explore data first" as an explicit option.

Phase 1.5 — Inventory what already exists

Each create sub-skill runs its own rigorous duplicate-check when invoked (signoz-creating-dashboards lists dashboards; signoz-creating-alerts paginates alert rules) — don't re-implement that here. The orchestration value is a single up-front sweep so you sequence around what exists instead of hitting collisions mid-build:

• Is the service reporting, and under what exact name?

(signoz_list_services.) This name is the resource filter every later phase reuses — pin it before anything else.

• Does a dashboard / alert / saved view already exist for it? If so,

plan to extend it — route to signoz-modifying-dashboards, signoz-managing-views, or signoz_update_alert — rather than standing up a parallel one. signoz-explaining-dashboards / signoz-explaining-alerts summarize anything you're inheriting.

• Is there a notification channel to reuse? (Carried into Phase 5.)

Paginate every inventory list until a match is found or all pages are exhausted. When a view's sourcePage is unknown, search traces, logs, metrics, and meter separately.

Learn just enough to decide extend-vs-create per artifact; leave the exhaustive listing to the sub-skills.

Phase 2 — Capture the SLI/SLO before designing anything

This is the gating artifact. Without it, alerts become "the metric crossed a line" instead of "the user-visible error budget is being burnt fast enough to matter."

Collect:

• SLI formula — good events / valid events. E.g. *successful

checkout requests / total checkout requests* (excluding health checks, synthetic probes, scanner traffic, and non-user long-poll endpoints).

• SLO target — e.g. 99.5% over a 30-day rolling window.
• Exclusions — what doesn't count as a valid event (synthetic

probes, scanner traffic, streaming endpoints).

• Error budget allowance — 1 - SLO, derived from the target. Remaining

budget additionally requires observed bad/valid events over the SLO window.

Carry these forward — they feed Phase 8 directly. (SLO/burn-rate background: signoz-searching-docs.)

Phase 3 — Explore what's actually there (RED + USE coverage)

signoz-generating-queries owns the exploration itself — the discover-before-querying rule, every discovery call (signoz_list_metrics, signoz_get_field_keys, signoz_get_field_values, signoz_get_service_top_operations), tool choice, and the no-data distinction. Drive it to inventory the data; don't restate its query mechanics here. (signoz-searching-docs covers the RED/USE methods and SigNoz field semantics if you need a refresher.)

The orchestration job is to map what it finds onto RED (Request rate, Errors, Duration) and USE (Utilization, Saturation, Errors), spot the gaps, and make the judgment calls generating-queries doesn't:

Coverage checklist (RED for services, USE for infra/resources):

• [ ] Rate — request/call rate per operation
• [ ] Errors — error count or rate per operation
• [ ] Duration — p50/p95/p99 latency from spans or histograms
• [ ] Utilization — CPU, memory, disk, queue depth, pool occupancy (only if you own the infra)
• [ ] Saturation — request queue, GC pause, connection pool wait
• [ ] Errors at the resource layer — disk failures, evictions, OOM kills

A missing signal (e.g. no error count) is an instrumentation gap — flag it before building around what's there.

Span metrics back alerts, not just panels. When generating-queries surfaces SigNoz's trace→metrics output (signoz_calls_total, signoz_latency_*, DB/external-call span metrics), prefer it as the RED source for both the dashboard and the burn-rate alerts: it's pre-aggregated and cheap, whereas raw trace aggregation over a long alert window is expensive and an unstable alert source. (Confirm exact metric/label names via discovery — don't hardcode.)

Render any counter by the volume you just measured, not by reflex. Not throughput-specific — it applies to every counter you'll graph (request rate, error counts, restarts, OOM kills, evictions), and low-volume errors and resource-layer events are the usual victims, not just requests. Per-second rate suits steady high-volume signals; where Phase 3 shows the count is low, bursty, or human-paced, a /sec graph renders as unreadable tiny decimals (0.03/s). Use the symptom, not a hard threshold: if per-second shows as fractions, pass that to Phase 6 as intent — render a per-interval increase count over a wider window (24h–7d), not a rate. (Gauges — CPU, memory, queue depth — are already absolute; this is a counter concern.) It's a deliberate tradeoff (increase rescales its y-axis with the selected range), so make it a conscious call, never a blanket default either way. signoz-creating-dashboards owns the rate-vs-increase mechanic.

Verify the infra→service join (the USE half). Don't assume service.name is present on host/container/k8s metrics — they often carry only host.name / k8s.pod.name / k8s.node.name, and k8s.deployment.name may differ from service.name. Have generating-queries confirm which identity attribute actually joins the infra metric to this workload, then scope USE panels by that confirmed attribute. If nothing joins, ask the user for the workload selector rather than guessing.

Discover downstream topology from traces, not the human's memory (Phase 1 Q1). The gotcha worth carrying: client spans are kind_string = 'Client' in the SigNoz v5 schema — not kind, not SPAN_KIND_CLIENT — so verify the field/value, then discover the populated dependency attributes with signoz_get_field_keys. Current examples include http_url, server.address, db.system, and db.operation; use only fields the tenant actually returns.

Multi-tenant note — if the service serves multiple tenants/regions, look for a stable non-PII tenant/region attribute (e.g. tenant.id, region, deployment.environment) — avoid raw URLs or user identifiers. Per-tenant breakdowns belong on the dashboard; aggregate-only views hide single-tenant outages.

SLI hygiene — confirm the events your SLI counts carry the right labels. For a broad denominator ("all requests to the service"), scanner traffic on /.env*, /.git* is real but should be excluded — it's not user traffic. The exclusion predicate should only subtract scanners and stay neutral otherwise, so use the bare negation — NOT REGEXP '/\\.(env|git|aws|ssh|well-known)' — not EXISTS AND …. SigNoz negative operators also match rows where the field is absent, so fieldless valid events (non-HTTP spans, internal ops) stay counted; let the SLI's own scope filter (service.name, http.route) define the event universe, not the scanner predicate. (Pair EXISTS with a negation only for the opposite intent — excluding a specific value among rows that have the field, e.g. service.name EXISTS AND service.name != 'redis'.) A well-scoped SLI already excludes scanner paths, so this mainly bites service-wide ratios. Keep a separate security view if you care about scanners.

Phase 4 — Classify labels by cardinality and cost

Before wiring labels into queries, classify each. Wrong classification is the #1 source of metric-bill explosions and dashboard unreadability.

| Class | Cardinality | Where it goes | |---|---|---| | Routing | `).

• Failed sub-flows (e.g. OAuth requests with errors).
• Correlation drill-downs — error logs carrying trace_id (a log spike

jumps straight to the trace) and slow/error traces that link back to their logs.

Two cross-phase constraints to pass along: every view carries the same environment/tenant filter as the dashboard (so prod and staging rows never mix), and the selected columns pass the same PII / no-identifiers gate as Phase 4.

Phase 8 — Alerts

signoz-creating-alerts owns every per-alert mechanic — channel resolution (create-first with a test message, name-not-UUID, secret handling), severity defaults, op/matchType, units, annotations, evalWindow/recoveryTarget, anomaly-rule shape, and the mandatory dry-run. Drive it once per alert and don't restate any of that here. (signoz-explaining-alerts reads a rule back; signoz-investigating-alerts does RCA once one fires.)

What this orchestration layer adds on top of that skill:

• SLI-bound alerts are burn-rate pairs, not static thresholds. A

static threshold on a ratio flaps; the burn-rate pair is the default for anything tied to the Phase 2 SLO.

• The burn-rate recipe (the one thing signoz-creating-alerts

can't derive without the SLO). Hand it the inputs, not a rule shape: the bad/total event definition (prefer the Phase-3 signoz_calls_total error/total split over raw trace aggregation), the long+short window lengths from Phase 5, and the firing target burnRate × (1 − SLO). Two judgment calls travel with those inputs — the long and short windows must be evaluated as one rule that ANDs both (two independent rules page on every short spike; if the SigNoz version can't express the short-window confirmation in a single rule, ship the long window alone and say so), and if no clean bad/total split exists in the ingested data, burn-rate isn't buildable yet — fall back to a baseline-tuned threshold rather than emit a broken rule. Let signoz-creating-alerts + signoz://alert/* pick the version-correct rule shape. Note that creating-alerts' "would have fired N times in the last 1h" calibration can't grade a 6h/3d burn-rate window — don't read a "fired 0 times" there as tuned; the Phase 9 re-tune-after-a-week loop is the real calibration.

• **Burn-rate only covers ratio SLIs — schedule the non-ratio failure

modes too.** A service that stops emitting, or whose traffic collapses, never burns error budget (the SLI ratio is undefined at zero volume), so the burn-rate pair pages no one. A complete setup adds a telemetry-presence / absent-data alert on the primary signal, and treats traffic/throughput as its own signal. Hand these intents to signoz-creating-alerts — it owns the rule shape (absent-data, anomaly, threshold) and the healthy-zero handling; the orchestration decision is only that they belong in the package.

• Batch + sequence. Resolve/confirm the channel (Phase 5 mapping)

before any rule references it, then create the alerts as a batch grouped by routing tier — not one rule per metric.

• Manual-TODO boundary. Inhibition rules, org-level notification

policies (usePolicy: true only opts into one), and maintenance windows are configured in the SigNoz UI / Terraform — these tools don't create them. Flag as TODO; never report them done.

Phase 9 — Hand off

Hand back to the human:

• Dashboard ID(s), view IDs, and alert IDs. Include a webUrl only when that

exact value was returned by an MCP tool; otherwise return the ID. Never construct a frontend URL from an ID or tenant hostname.

• Open questions you couldn't resolve — hand them over as explicit

TODOs.

• Baselines that may need tuning (e.g. "tool error rate is 5.1% today;

burn-rate alert tuned for 99% SLO will fire occasionally — re-tune after a week of data").

What to avoid

• Inventing channel names or webhook URLs. Ask.
• Static thresholds for SLI-bound alerts. Burn-rate pairs are the

modern default; static thresholds for ratios produce flap.

• Wrong-sizing dashboards. Don't cram 18 panels where 12 will do,

and don't scatter four 2-panel boards where one would read better — split by category, then right-size each. Sprawl cuts both ways.

• **Skipping dry-runs

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: SigNoz
• Source: SigNoz/agent-skills
• License: MIT
• Homepage: https://signoz.io/docs/ai/agent-skills

Install and usage instructions live in the source repository linked above.