# Signoz Investigating Alerts

> >

- **Type:** Skill
- **Install:** `agentstack add skill-therapys-dotagents-signoz-investigating-alerts`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [therapys](https://agentstack.voostack.com/s/therapys)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [therapys](https://github.com/therapys)
- **Source:** https://github.com/therapys/dotagents/tree/main/skills/signoz-investigating-alerts

## Install

```sh
agentstack add skill-therapys-dotagents-signoz-investigating-alerts
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Alert Investigate

Diagnose why a SigNoz alert fired: correlate the alert's own signal with
neighbor signals around the fire window and surface a ranked list of likely
causes with evidence. Companion to `signoz-explaining-alerts` (explain
decodes the rule statically; investigate diagnoses a specific incident).

## Prerequisites

Needs SigNoz MCP tools (`signoz_get_alert`, `signoz_get_alert_history`,
`signoz_execute_builder_query`, `signoz_query_metrics`,
`signoz_search_traces`, `signoz_search_logs`, `signoz_get_trace_details`,
etc.). Confirm `signoz_*` tools are available before starting; if not, run
`signoz-mcp-setup` to initialize/repair the MCP connection. Without the
server there is no way to ground the analysis.

## When to use

Use when the user wants to:
- Understand why a specific alert fired.
- Find the root cause of a recent incident triggered by an alert.
- Correlate the alert's signal with related metrics, traces, and logs.
- Distinguish "real signal" fires from flapping or threshold-mistuning.

Do NOT use when the user wants to:
- Understand what an alert monitors → `signoz-explaining-alerts`.
- Create a new alert → `signoz-creating-alerts`.
- Modify an alert (raise threshold, add hysteresis) → call
  `signoz_update_alert` directly.
- Run a free-form investigation with no alert anchor →
  `signoz-generating-queries`.

## Required inputs

| Input | Required | Source if missing |
|---|---|---|
| Alert identifier (rule ID or name) | yes | `$ARGUMENTS[0]` or recent context |
| Time window | no | default to most recent fire from `signoz_get_alert_history` |

Fuzzy alert name (best-effort, read-only):
1. Call `signoz_list_alert_rules`, paginate, fuzzy-match the name.
2. State the interpretation ("Investigating fire of 'High Error Rate -
   Checkout' (id 42) at 14:32 UTC. If you meant a different alert or fire,
   tell me.") and proceed.

If no firing transition exists in the queried lookback, **stop** — there is
nothing to investigate. Respond:
> "Alert '[name]' has not fired in the last 7d, so there is no fire window
> to investigate. Use `signoz-explaining-alerts` to walk through the rule,
> or check whether the alert is enabled."

## Workflow

Three tiers with strict early-stop gates. Tier 1 always runs; Tier 2 runs
only if Tier 1 confirms a real fire; Tier 3 runs only if Tier 2 surfaces
correlated anomalies. Skipping the gates fires hundreds of unnecessary
trace/log queries on quiet alerts.

### Step 1: Resolve alert + fire window (Tier 0)

1. Resolve the alert id via `signoz_list_alert_rules` (paginated) if not
   given.
2. Call `signoz_get_alert` for the full rule config (query, threshold,
   resource scope the alert evaluated).
3. Call `signoz_get_alert_history` with `timeRange: "7d"`, `order: "desc"`,
   and `state` omitted so the timeline includes firing and inactive
   transitions. Paginate only when `data.nextCursor` exists (completeness
   note reports `hasMore: true`): pass it as `cursor`, replace `timeRange`
   with the note's resolved absolute `start`/`end`, and preserve the same
   state/filter (including omission) and order. Stop when `nextCursor` is
   absent / `hasMore: false`; never use `offset` or page fullness. If a
   later intentional filter means "resolved"/"recovered", use `inactive`.
   Enum: `inactive|pending|recovering|firing|nodata|disabled`; `recovering`
   is a transient keep-firing state, not resolution. Pattern analysis needs
   the complete transition set. Rows are emitted per label-group
   `fingerprint`; do not interleave them. From the response:
   - **Build rule-wide incident windows** from distinct rows where
     `overallStateChanged: true`: `overallState: "firing"` opens an incident;
     the next `overallState: "inactive"` closes it. Deduplicate matching
     timestamps and sort by `unixMilli` ascending before pairing. Default to
     the most recent incident unless `$ARGUMENTS[1]` selects another.
   - **Partition affected series by `fingerprint`** and retain each row's
     labels. Use only `stateChanged: true` rows to decide when that group
     fired/resolved and which group scopes Tier 1–3 queries.
   - **Note the fire pattern** from rule-wide transitions or one named
     fingerprint:
     - `one-off` → single fire with a long quiet period before/after.
     - `sustained` → fires that stayed firing for ≥ 1 evaluation cycle.
     - `flapping` → ≥ 3 fires within a 1h window, alternating fire/resolve.
     - `recurring` → fires at regular intervals (cron-like, e.g. hourly).
   - Never infer flapping from different fingerprints. The pattern guides
     tiers 2/3.

### Step 2: Tier 1 — what fired and how hard (always runs)

Establishes the fire is real (vs. transient threshold tickle or flap) and
quantifies magnitude.

1. Re-run the alert's primary query over `[fire_start - 30m, fire_start +
   30m]` via `signoz_execute_builder_query` for the alert's stored builder,
   formula, PromQL, or ClickHouse envelope. Preserve positive bounds/order
   so Tier 1 reproduces the stored alert. If a formula input is below 10000,
   record truncation risk and compare at 10000 before ruling groups out. For
   omissions, use 10000 on formula-input `builder_query` leaves and 100 on
   standalone/formula results. Find leaves from every formula expression
   (including `disabled: true` formulas), following references through the
   dependency graph. This walk sets comparison bounds only; it does not prove
   deterministic formula-to-formula order. Use v5 `order`: `__result desc`
   for metrics/formulas or primary aggregation desc for logs/traces, never
   dashboard `orderBy`. Time-series top-N ranks over the whole window and may
   omit a short-lived local spike.
2. Compute:
   - **Peak value** during the fire window.
   - **Threshold breach magnitude**: `(peak - threshold) / threshold * 100`
     for "above" alerts, inverted for "below".
   - **Fire duration**: the rule's overall firing→inactive interval, or the
     selected fingerprint's interval for a group-scoped investigation. Say
     which.
   - **Pre-fire baseline**: average in the 30m before fire start.
3. **Early-stop gate**: if breach magnitude `
   AND `has_error = true`, cap at top 20. Group by `name` and
   `status_message`; surface the sample's top 3 with one trace ID each (do
   not treat a 20-row sample count as full-window frequency). Optionally call
   `signoz_get_trace_details` for span attributes — pass the search row's
   `trace_id` as `traceId` **plus the same absolute fire-window `start` and
   `end`**, else the 6h default misses older incidents.
2. **Logs**: call `signoz_search_logs` with `` AND
   `severity_text IN ('ERROR', 'FATAL')`, cap at top 20 most recent. Group by
   `body` pattern (or `exception.type` if present); surface the top 3
   distinct messages with counts.
3. **Cross-reference**: do the traces and logs point at the same downstream
   service, dependency, or code path? If so, that becomes the leading
   hypothesis.

See `references/baseline-comparison.md` for query templates that pair
fire-window and baseline-window calls cleanly.

### Step 5: Structured output

Use this exact section order. Lead with the TL;DR (engineers under pressure
scan the top and stop once they have what they need). Every claim cites the
MCP query that produced it; no generic "check logs / verify connectivity"
filler.

**1. TL;DR** — one or two sentences max: leading hypothesis, overall
confidence, blast radius, single most useful next action. Example:
> "checkoutservice error rate hit 12.4% (threshold 5%) for 8m at 14:32 UTC;
> most likely cause is payments-api timing out (high confidence). Open trace
> `7af3a09b…` to see the failing call."

If no hypothesis reaches medium confidence, lead with "No clear root cause
found." rather than a dressed-up low-confidence guess.

**2. What fired** — alert (id, name), fire window (absolute UTC + relative),
peak magnitude ("error rate hit 12.4% vs. 5% threshold, 148% over"), fire
duration, fire pattern (`one-off`/`sustained`/`flapping`/`recurring`/
`marginal`).

**3. Investigation trail** — scannable list of what was checked, ✅ for
confirmed and ❌ for ruled out, each with a one-line finding, e.g.:
- ✅ Tier 1: peak error rate 12.4%, fire was real (not marginal).
- ✅ Tier 2: payments error rate +8900%, p99 +1180%; downstream cascade.
- ❌ CPU / memory pressure: flat through the fire window.
- ✅ Tier 3: 30 error traces all hit payments-api, same message.

**4. Likely causes** (ranked, max 3) — each has:
- **Hypothesis**: one specific sentence. Bad: "service is unhealthy". Good:
  "checkout is timing out on calls to payments-api".
- **Evidence**: supporting numbers from tiers 1/2/3 with the underlying query
  inline (neighbor signal, delta vs baseline, trace/log pattern) so the user
  can re-run it.
- **Confidence**: `high` requires ≥2 of {temporal precedence, topology /
  dependency edge, shared service or entity, correlated metric/log/trace
  evidence, recent deploy or config change}. `medium` is one tier's evidence
  with ≥1 of those. `low` is a single signal with no corroboration — label it
  a "co-occurring signal," not a cause.

If only Tier 1 ran (marginal / no neighbor anomalies), output fewer
hypotheses at `low` confidence and call out the limitation.

**5. Ruled out** — short but explicit: candidates the evidence eliminated and
the one-line reason. Skip only if there is nothing meaningful; if you
considered and dropped something, say so here.

**6. Suggested next steps** — concrete, SigNoz-native handles the user can
act on immediately:
- Specific trace/dashboard/alert to open ("open trace `7af3a09b…` in the
  SigNoz UI").
- Specific query to run with `signoz-generating-queries`: paste the exact
  filter and time window.
- "Tune this alert" if marginal: name the field (`matchType`, `target`,
  `recoveryTarget`) and the change to make via `signoz_update_alert`.
- "Open an incident" / "page the owning team" if the cause is cross-service.

No generic filler ("verify connectivity", "check dashboards").

**Mirror up to 3** of these next steps as host follow-up intents: the most
actionable, alert-scoped ones. Keep the rest in the report prose (the chip
surface is capped; the prose is not).

## Out of scope (v1)

- **Deploy / config-change correlation**: SigNoz MCP exposes no deployments
  tool; do not fabricate one. Surface a user-mentioned deploy as context, but
  don't claim it caused the fire without signal evidence.
- **Cross-service blast-radius walking** (downstream callers): out of scope
  to keep context bounded.
- **Long-horizon historical baselines**: Tier 2 compares to one prior-day
  window, not weekly/monthly seasonality. If asked "is this normal for a
  Friday afternoon", suggest an anomaly alert (`signoz-creating-alerts` with
  `anomaly_rule`).

## Guardrails

- **Early-stop gates are mandatory**, not optional optimizations — skipping
  them explodes context.
- **Anchor every claim to a real MCP query result**; no speculation, no
  fabricated identifiers (trace IDs, span names, rule IDs, channel/deploy IDs
  must all come from real responses). If evidence is missing, lower
  confidence and say so.
- **Correlation ≠ causation**: only call something a cause with ≥2 converging
  signals (see Confidence in Step 5); otherwise it's a "co-occurring signal."
- **Report uncertainty honestly**: if nothing reaches medium confidence, the
  answer is "No clear root cause found" — false positives waste incident time
  more than false negatives.
- **Keep it a triage card, not a postmortem** — every section must add
  evidence the reader can't already see in the alert payload; drop generic
  advice.
- **Prefer resource-attribute filters** in every drill-down query (SigNoz MCP
  guideline; directly affects query speed at scale).
- **Read-only**: never call `signoz_update_alert`; surface tuning as a
  next-step recommendation.
- **State fire windows in UTC + relative** ("14:32 UTC, 2h ago") so
  autonomous and interactive consumers agree on the window.

## Example (full 3-tier run)

**User:** "Why did the checkout error rate alert fire?"

**Agent:**
1. Resolves alert: "High Error Rate - Checkout" (id 42).
2. `signoz_get_alert_history` → most recent fire 2h ago at 14:32 UTC,
   sustained 8m, single fire (not flapping).
3. **Tier 1**: re-runs error-rate formula over `[14:02, 15:02]`. Peak 12.4%
   (vs 5% threshold, 148% over), pre-fire baseline 0.3%. Real fire.
4. **Tier 2** (`service.name = checkout`): p99 4.1s vs 320ms (+1180%);
   throughput −42%; downstream `payments` error rate 18% vs 0.2% (+8900%);
   CPU/memory flat.
5. **Tier 3**: error traces top op `POST /checkout/submit`, message "context
   deadline exceeded calling payments-api", 30 traces same downstream URL;
   matching "payments client timeout" logs, 142 occurrences.
6. **Output**:

   > **TL;DR**: checkoutservice error rate hit 12.4% (threshold 5%) for 8m at
   > 14:32 UTC. Most likely cause: payments-api timing out (high confidence;
   > converging trace + log + neighbor evidence). Open trace `7af3a09b…`.

   - **What fired**: alert 42 fired 2h ago at 14:32 UTC, sustained 8m; error
     rate peaked 12.4% (148% over threshold).
   - **Investigation trail**:
     - ✅ Tier 1: peak 12.4% vs 5%, pre-fire baseline 0.3%. Real fire.
     - ✅ Tier 2: payments error rate +8900%, p99 +1180%, throughput −42%.
     - ❌ CPU / memory on checkout: flat.
     - ✅ Tier 3: 30 error traces all hit payments-api with `context deadline
       exceeded`; 142 matching timeout logs.
   - **Likely causes** (high): payments errors cascading into checkout.
     Evidence converges across topology (checkout → payments edge), temporal
     precedence (payments errors lead checkout p99), shared entity (every
     error trace targets the payments-api URL).
   - **Ruled out**: checkout-side resource pressure (CPU/memory flat);
     upstream traffic spike (throughput dropped, didn't spike).
   - **Next steps**: open trace `7af3a09b…`; if payments has its own alert,
     run `signoz-explaining-alerts` on it, else `signoz-generating-queries`
     for payments error rate over the last 4h.

Same structure applies to other patterns, differing only in where the tiers
stop. A **flapping / marginal** fire (e.g. CPU hovering 5% over an 80%
threshold, `matchType = at_least_once`) early-stops after Tier 1 → output is
a low-confidence "threshold tuning, not an incident" with a `matchType` →
`on_average` / `recoveryTarget` fix. A **log-volume** fire (no service
filter, `groupBy service.name`) resolves the firing series from history, then
runs all three tiers on that service (e.g. Tier 2 shows collapsed CPU/memory
+ pod restarts, Tier 3 shows "OOMKilled restarting" logs → high-confidence
OOM cause, check container memory limits).

## Additional resources

- `references/neighbor-signals.md`: resource type (service / host / k8s) →
  neighbor signals to pull in Tier 2.
- `references/baseline-comparison.md`: query templates pairing fire-window and
  baseline-window calls, including `signoz_execute_builder_query` format.
- `signoz-explaining-alerts`: decode the rule before investigating.
- `signoz-generating-queries`: ad-hoc follow-up queries on the same scope.

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [therapys](https://github.com/therapys)
- **Source:** [therapys/dotagents](https://github.com/therapys/dotagents)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-therapys-dotagents-signoz-investigating-alerts
- Seller: https://agentstack.voostack.com/s/therapys
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.