# Observability Service Health

> >

- **Type:** Skill
- **Install:** `agentstack add skill-elastic-agent-skills-service-health`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [elastic](https://agentstack.voostack.com/s/elastic)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** Apache-2.0
- **Upstream author:** [elastic](https://github.com/elastic)
- **Source:** https://github.com/elastic/agent-skills/tree/main/plugins/observability/skills/service-health

## Install

```sh
agentstack add skill-elastic-agent-skills-service-health
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# APM Service Health

Assess APM service health using [Observability APIs](https://www.elastic.co/docs/solutions/observability/apis),
**ES|QL** against APM indices, Elasticsearch APIs, and (for correlation and APM-specific logic) the Kibana repo. Use
SLOs, firing alerts, ML anomalies, throughput, latency (avg/p95/p99), error rate, and dependency health.

## Where to look

- **Observability APIs** ([Observability APIs](https://www.elastic.co/docs/solutions/observability/apis)): Use the
  **SLOs API** ([Stack](https://www.elastic.co/docs/api/doc/kibana/group/endpoint-slo) |
  [Serverless](https://www.elastic.co/docs/api/doc/serverless/group/endpoint-slo)) to get SLO definitions, status, burn
  rate, and error budget. Use the **Alerting API**
  ([Stack](https://www.elastic.co/docs/api/doc/kibana/group/endpoint-alerting) |
  [Serverless](https://www.elastic.co/docs/api/doc/serverless/group/endpoint-alerting)) to list and manage alerting
  rules and their alerts for the service. Use **APM annotations API** to create or search annotations when needed.
- **ES|QL and Elasticsearch:** Query `traces*apm*,traces*otel*` and `metrics*apm*,metrics*otel*` with **ES|QL** (see
  [Using ES|QL for APM metrics](#using-esql-for-apm-metrics)) for throughput, latency, error rate, and dependency-style
  aggregations. Use Elasticsearch APIs (e.g. `POST _query` for ES|QL, or Query DSL) as documented in the Elasticsearch
  repo for indices and search.
- **APM Correlations:** Run the **apm-correlations** script to get attributes that correlate with high-latency or failed
  transactions for a given service. It tries the Kibana internal APM correlations API first, then falls back to
  Elasticsearch significant_terms on `traces*apm*,traces*otel*`. See
  [APM Correlations script](#apm-correlations-script).
- **Infrastructure:** Correlate via **resource attributes** (e.g. `k8s.pod.name`, `container.id`, `host.name`) in
  traces; query infrastructure or metrics indices with ES|QL/Elasticsearch for CPU and memory. **OOM** and **CPU
  throttling** directly impact APM health.
- **Logs:** Use **ES|QL** or Elasticsearch search on log indices filtered by `service.name` or `trace.id` to explain
  behavior and root cause.
- **Observability Labs:** [Observability Labs](https://www.elastic.co/observability-labs) and
  [APM tag](https://www.elastic.co/observability-labs/blog/tag/apm) for patterns and troubleshooting.

## Health criteria

Synthesize health from all of the following when available:

| Signal                | What to check                                                             |
| --------------------- | ------------------------------------------------------------------------- |
| **SLOs**              | Burn rate, status (healthy/degrading/violated), error budget.             |
| **Firing alerts**     | Open or recently fired alerts for the service or dependencies.            |
| **ML anomalies**      | Anomaly jobs; score and severity for latency, throughput, or error rate.  |
| **Throughput**        | Request rate; compare to baseline or previous period.                     |
| **Latency**           | Avg, p95, p99; compare to SLO targets or history.                         |
| **Error rate**        | Failed/total requests; spikes or sustained elevation.                     |
| **Dependency health** | Downstream latency, error rate, availability (ES\|QL, APIs, Kibana repo). |
| **Infrastructure**    | CPU usage, memory; OOM and CPU throttling on pods/containers/hosts.       |
| **Logs**              | App logs filtered by service or trace ID for context and root cause.      |

Treat a service as **unhealthy** if SLOs are violated, critical alerts are firing, or ML anomalies indicate severe
degradation. Correlate with infrastructure (OOM, CPU throttling), dependencies, and logs (service/trace context) to
explain _why_ and suggest next steps.

## Using ES|QL for APM metrics

When querying APM data from Elasticsearch (`traces*apm*,traces*otel*`, `metrics*apm*,metrics*otel*`), use **ES|QL by
default** where available.

- **Availability:** ES|QL is available in **Elasticsearch 8.11+** (technical preview; GA in 8.14). It is **always
  available** in
  [Elastic Observability Serverless Complete tier](https://www.elastic.co/docs/solutions/observability/observability-serverless-feature-tiers).
- **Scoping to a service:** Always filter by `service.name` (and `service.environment` when relevant). Combine with a
  time range on `@timestamp`:

```esql
WHERE service.name == "my-service-name" AND service.environment == "production"
  AND @timestamp >= "2025-03-01T00:00:00Z" AND @timestamp "`.
- **Performance:** Add `LIMIT n` to cap rows and token usage. Prefer coarser `BUCKET(@timestamp, ...)` (e.g. 1 hour)
  when only trends are needed; finer buckets increase work and result size.

## APM Correlations script

When only a **subpopulation** of transactions has high latency or failures, run the **apm-correlations** script to list
attributes that correlate with those transactions (e.g. host, service version, pod, region). The script tries the Kibana
internal APM correlations API first; if unavailable (e.g. 404), it falls back to Elasticsearch significant_terms on
`traces*apm*,traces*otel*`.

```bash
# Latency correlations (attributes over-represented in slow transactions)
node skills/observability/service-health/scripts/apm-correlations.js latency-correlations --service-name  [--start ] [--end ] [--last-minutes 60] [--transaction-type ] [--transaction-name ] [--space ] [--json]

# Failed transaction correlations
node skills/observability/service-health/scripts/apm-correlations.js failed-correlations --service-name  [--start ] [--end ] [--last-minutes 60] [--transaction-type ] [--transaction-name ] [--space ] [--json]

# Test Kibana connection
node skills/observability/service-health/scripts/apm-correlations.js test [--space ]
```

**Environment:** `KIBANA_URL` and `KIBANA_API_KEY` (or `KIBANA_USERNAME`/`KIBANA_PASSWORD`) for Kibana; for fallback,
`ELASTICSEARCH_URL` and `ELASTICSEARCH_API_KEY`. Use the same time range as the investigation.

## Workflow

```text
Service health progress:
- [ ] Step 1: Identify the service (and time range)
- [ ] Step 2: Check SLOs and firing alerts
- [ ] Step 3: Check ML anomalies (if configured)
- [ ] Step 4: Review throughput, latency (avg/p95/p99), error rate
- [ ] Step 5: Assess dependency health (ES|QL/APIs / Kibana repo)
- [ ] Step 6: Correlate with infrastructure and logs
- [ ] Step 7: Summarize health and recommend actions
```

### Step 1: Identify the service

Confirm service name and time range. Resolve the service from the request; if multiple are in scope, target the most
relevant. Use ES|QL on `traces*apm*,traces*otel*` or `metrics*apm*,metrics*otel*` (e.g.
`WHERE service.name == ""`) or Kibana repo APM routes to obtain service-level data. If the user has not provided
the time range, assume last hour.

### Step 2: Check SLOs and firing alerts

**SLOs:** Call the **SLOs API** to get SLO definitions and status for the service (latency, availability),
healthy/degrading/violated, burn rate, error budget. **Alerts:** For active APM alerts, call
`/api/alerting/rules/_find?search=apm&search_fields=tags&per_page=100&filter=alert.attributes.executionStatus.status:active`.
When checking one service, include both rules where `params.serviceName` matches the service and rules where
`params.serviceName` is absent (all-services rules). Do not query `.alerts*` indices for active-state checks. Correlate
with SLO violations or metric changes.

### Step 3: Check ML anomalies

If ML anomaly detection is used, query ML job results or anomaly records (via Elasticsearch ML APIs or indices) for the
service and time range. Note high-severity anomalies (latency, throughput, error rate); use anomaly time windows to
narrow Steps 4–5.

### Step 4: Review throughput, latency, and error rate

Use **ES|QL** against `traces*apm*,traces*otel*` or `metrics*apm*,metrics*otel*` for the service and time range to get
**throughput** (e.g. req/min), **latency** (avg, p95, p99), **error rate** (failed/total or 5xx/total). Example:
`FROM traces*apm*,traces*otel* | WHERE service.name == "" AND @timestamp >= ... AND @timestamp  [--start ...] [--end ...]`
  to get correlated attributes. Filter by those attributes and fetch trace samples or errors to confirm root cause. See
  [APM Correlations script](#apm-correlations-script).
- **Infrastructure:** Use **resource attributes** from traces (e.g. `k8s.pod.name`, `container.id`, `host.name`) and
  query infrastructure/metrics indices with **ES|QL** or Elasticsearch for **CPU** and **memory**. **OOM** and **CPU
  throttling** directly impact APM health; correlate their time windows with APM degradation.
- **Logs:** Use **ES|QL** or Elasticsearch on log indices with `service.name == ""` or
  `trace.id == ""` to explain behavior and root cause (exceptions, timeouts, restarts).

### Step 7: Summarize and recommend

State health (**healthy** / **degraded** / **unhealthy**) with reasons; list concrete next steps.

## Examples

### Example: ES|QL for a specific service

Scope with `WHERE service.name == ""` and time range. Throughput and error rate (1-hour buckets; `LIMIT`
caps rows and tokens):

```esql
FROM traces*apm*,traces*otel*
| WHERE service.name == "api-gateway"
  AND @timestamp >= "2025-03-01T00:00:00Z" AND @timestamp "` and time
  range to get application logs (errors, warnings, restarts) in the service context.
- **By trace ID:** When investigating a specific request, take the `trace.id` from the APM trace and filter logs by
  `trace.id == ""` (or equivalent field in your log schema). Logs with that trace ID show the full request
  path and help explain failures or latency.

## Guidelines

- Use **Observability APIs** ([SLOs API](https://www.elastic.co/docs/api/doc/kibana/group/endpoint-slo),
  [Alerting API](https://www.elastic.co/docs/api/doc/kibana/group/endpoint-alerting)) and **ES|QL** on
  `traces*apm*,traces*otel*`/`metrics*apm*,metrics*otel*` (8.11+ or Serverless), filtering by `service.name` (and
  `service.environment` when relevant). For active APM alerts, call
  `/api/alerting/rules/_find?search=apm&search_fields=tags&per_page=100&filter=alert.attributes.executionStatus.status:active`.
  When checking one service, evaluate both rule types: rules where `params.serviceName` matches the target service, and
  rules where `params.serviceName` is absent (all-services rules). Treat either as applicable to the service before
  declaring health. Do not query `.alerts*` indices when determining currently active alerts; use the Alerting API
  response above as the source of truth. For APM correlations, run the **apm-correlations** script (see
  [APM Correlations script](#apm-correlations-script)); for dependency/service-map data, use ES|QL or Kibana repo route
  handlers. For Elasticsearch index and search behavior, see the **Elasticsearch** APIs in the Elasticsearch repo.
- Always use the **user's time range**; avoid assuming "last 1 hour" if the issue is historical.
- When SLOs exist, anchor the health summary to SLO status and burn rate; when they do not, rely on alerts, anomalies,
  throughput, latency, error rate, and dependencies.
- When analyzing **only application metrics ingested via OpenTelemetry**, use the ES|QL **TS** (time series) command for
  efficient metrics queries. The TS command is available in **Elasticsearch 9.3+** and is **always available** in
  Elastic Observability Serverless.
- Summary: one short health verdict plus bullet points for evidence and next steps.

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [elastic](https://github.com/elastic)
- **Source:** [elastic/agent-skills](https://github.com/elastic/agent-skills)
- **License:** Apache-2.0

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-elastic-agent-skills-service-health
- Seller: https://agentstack.voostack.com/s/elastic
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.