# Observability K8s Investigation > > - **Type:** Skill - **Install:** `agentstack add skill-elastic-agent-skills-k8s-investigation` - **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior - **Seller:** [elastic](https://agentstack.voostack.com/s/elastic) - **Installs:** 0 - **Category:** [Cloud & Infrastructure](https://agentstack.voostack.com/c/cloud-infrastructure) - **Latest version:** 0.1.0 - **License:** Apache-2.0 - **Upstream author:** [elastic](https://github.com/elastic) - **Source:** https://github.com/elastic/agent-skills/tree/main/plugins/observability/skills/k8s-investigation ## Install ```sh agentstack add skill-elastic-agent-skills-k8s-investigation ``` Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent. ## About # Kubernetes Investigation Diagnose Kubernetes issues using OTel telemetry collected via EDOT (Elastic Distribution of OpenTelemetry) and the kube-stack collector. Correlate cluster state, pod runtime metrics, K8s events, application logs, and APM to identify root cause across the workload, node, and control-plane layers. ## Scope **In scope:** OTel-receiver-namespaced indices (`metrics-kubeletstatsreceiver.otel-*`, `metrics-k8sclusterreceiver.otel-*`, `logs-k8seventsreceiver.otel-*`, `logs-k8sobjectsreceiver.otel-*`) and OTel semantic conventions (`k8s.pod.name`, `k8s.namespace.name`, `k8s.container.restarts`). **Out of scope:** - The legacy Elastic Agent Kubernetes integration (`metrics-kubernetes.*`, `logs-kubernetes.*`, `kubernetes.*` fields). Being deprecated — do not author queries against these paths. - APM-layer analysis (service SLO breaches, transaction error rates, upstream dependency health). Different domain — once a K8s root cause is ruled in or out, APM investigation continues outside this skill. - Cluster provisioning, capacity planning, cost optimization. Different domain. ## Guidelines These apply to every investigation. When in doubt, re-read them before writing the synthesis. **Absence of evidence is not evidence. Do not confabulate from empty results.** If log queries return 0 rows, logs are likely not collected or the pod has no recent lines — this does _not_ mean "dependency unavailable" or any other specific failure mode. Report `no_logs_available` and weight remaining signals accordingly. **Empty dependency data ≠ upstream healthy.** Services without APM instrumentation (load generators, workers) emit no destination metrics. Report `insufficient_dependency_data`, not "upstreams OK." **Co-symptoms are not causes.** Two services degrading simultaneously usually share an upstream, not a causal link. Only attribute causation when (a) one service's degradation clearly precedes the other's, and (b) the delta is large (>5× error rate, >3× latency). **OOMKilled ≠ memory leak by default.** The limit may simply be undersized for the workload's working set. Compare against a 7-day baseline at the same hour-of-day before claiming a leak. **Error-termination ≠ application bug by default.** Check `k8s.pod.cpu_limit_utilization` first. CFS throttling driving liveness probe timeouts is the most common misdiagnosis in this space. **Average CPU hides throttling.** A pod can look healthy at 40–60% average `cpu_limit_utilization` while being throttled severely at p99. Linux enforces CPU limits in 100ms periods; bursty workloads hit quota mid-period and stall. Look at max and p95, not just average. **Restart count is boolean, not a counter.** `k8s.container.restarts` is pulled directly from the K8s API and may be pruned by the kubelet at any time, so the absolute value is unreliable. Treat it as `== 0` (no recent restarts) vs `> 0` (recently restarting); do not derive backoff timing or "linear vs exponential" patterns from it. Confirm the restart pattern via K8s `Killing` / `BackOff` events instead. **Prefer to report uncertainty over manufacturing confidence.** If the evidence is ambiguous, the synthesis should say so. Competing hypotheses are a valid output. ## Indices and fields ### Where to look | Signal | Index pattern | Use | | --------------------- | --------------------------------------------------- | ------------------------------------------------------------------- | | Pod/container runtime | `metrics-kubeletstatsreceiver.otel-*` | CPU, memory, network, filesystem. Utilization ratios. | | Cluster state | `metrics-k8sclusterreceiver.otel-*` | Restarts, phase, last-terminated reason, HPA, quota, node condition | | K8s events | `logs-k8seventsreceiver.otel-*` | Killing, BackOff, FailedScheduling, Evicted, image pull events | | K8s object snapshots | `logs-k8sobjectsreceiver.otel-*` | Deployment/service/configmap state over time | | Application logs | `logs-*.otel-*` | `body.text`, `severity_text`, filtered by `k8s.pod.name` | | APM | `traces-*.otel-*`, `metrics-service_*.otel-default` | Correlate via `service.name` + K8s resource attrs | | ML anomalies | `.ml-anomalies-*` | Memory-growth, restart-rate, throttle jobs (if configured) | ### Key fields Flat OTel paths work in ES|QL. Prefer the flat form for readability; the nested `resource.attributes.*` form is for raw log documents only. | Field | Index | What it is | | ------------------------------------------------ | --------------------------- | ------------------------------------------------------- | | `k8s.pod.name` | all k8s | Pod name | | `k8s.namespace.name` | all k8s | Namespace | | `k8s.container.name` | all k8s | Container within pod | | `k8s.deployment.name` | k8sclusterreceiver + others | Parent deployment | | `k8s.pod.phase` | k8sclusterreceiver | Pending=1/Running=2/Succeeded=3/Failed=4/Unknown=5 | | `k8s.container.restarts` | k8sclusterreceiver | Total container restart count | | `k8s.container.status.last_terminated_reason` | k8sclusterreceiver | `OOMKilled`, `Error`, `Completed`, `ContainerCannotRun` | | `k8s.pod.status_reason` | k8sclusterreceiver | Pod-level reason (`Evicted`, `NodeLost`) | | `k8s.pod.memory_limit_utilization` | kubeletstatsreceiver | 0.0–1.0+ (can exceed 1 transiently before OOM) | | `k8s.pod.cpu_limit_utilization` | kubeletstatsreceiver | 0.0–N (frequently >1 under CFS throttling) | | `k8s.pod.memory.usage` / `.working_set` | kubeletstatsreceiver | Bytes | | `k8s.node.condition_memory_pressure` | k8sclusterreceiver | 1 = pressure, 0 = ok | | `k8s.node.condition_ready` | k8sclusterreceiver | 0 = NotReady | | `k8s.hpa.current_replicas` / `.desired_replicas` | k8sclusterreceiver | HPA state | | `attributes.k8s.event.reason` | k8seventsreceiver | Event reason (filter on this) | | `body.text` | k8seventsreceiver / logs | Event message / log message | | `k8s.object.name` | k8seventsreceiver | involvedObject name (log attribute, use flat form) | ### Field availability Several fields above are off by default in stock kube-stack collectors and require explicit configuration. Verify presence before relying on them; if absent, fall back as noted and call out the substitution in the synthesis. | Field | Why it might be missing | Fall-back | | ------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | | `k8s.container.status.last_terminated_reason` | Optional metric in k8sclusterreceiver; gated behind `metrics_collected.metadata` config. | Infer from K8s `Killing` / `OOMKilling` events in `logs-k8seventsreceiver.otel-*` and exit codes in app logs. | | `k8s.pod.status_reason` | Same — optional metric on k8sclusterreceiver. | Infer from events: `Evicted`, `NodeLost`, `Preempted`. | | `k8s.pod.cpu_limit_utilization` / `memory_limit_utilization` | Only emitted when the pod has the corresponding limit set, and the kubeletstatsreceiver metric is enabled. | Compute manually as `k8s.pod.cpu.usage / ` from k8sclusterreceiver, or use absolute usage trending against a baseline. | | `k8s.node.condition_memory_pressure` | Gated behind k8sclusterreceiver `node_conditions_to_report` (default omits this). | Compare `k8s.node.memory.usage` against `k8s.node.allocatable_memory`, or look for `Evicted` events on the node. | If a fall-back is used, note it in the synthesis (e.g. `(via memory.usage; limit_utilization not collected)`) so the reader knows the signal is indirect. ## ES|QL gotchas Before writing queries, know these. Each of them silently produces wrong answers rather than failing loudly. **`VALUES()` returns scalar for single distinct value, array for multiple.** Templating that assumes array shape (e.g. `| first`) extracts the first character of the string when scalar. Use `MV_FIRST(VALUES(...))` or handle both. **`PERCENTILE` does not work on OTel `histogram` type** (as of 8.15). For APM duration percentiles, use `AVG` on the `aggregate_metric_double` summary field (`AVG(transaction.duration.summary)` divides sum by value_count). For true percentiles, fall back to Kibana Query DSL. **`COUNT(agg_metric_double)` returns `value_count` (events), not doc count.** `SUM(field)` gives the sum component; `AVG(field)` gives sum/value_count. Do not use `SUM(transaction.duration.summary)` as an event-count proxy — it returns total duration. **K8s metrics use flat OTel field paths in ES|QL.** `k8s.pod.name`, not `resource.attributes.k8s.pod.name`. The nested form is for raw log documents. ## Failure-mode taxonomy Vocabulary for classification, not a decision tree. Use the pivotal-signal column to recognize which mode you're looking at; use "Investigate" to know what else should corroborate. ### Workload layer | Mode | Pivotal signal | Investigate | | ----------------------------------- | -------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **OOMKilled** | `last_terminated_reason == "OOMKilled"` + `memory_limit_utilization → 1.0` | Monotonic rise (leak) vs. load-driven spike? Compare current trend to 7-day baseline. Check heap metrics (JVM, Go, Node) for GC pressure. | | **CPU throttling → Error exit** | `cpu_limit_utilization > 1.0` + `last_terminated_reason == "Error"` | Liveness/readiness probe timeouts from CFS throttling. Average CPU can look fine (40–60%) while p99 throttle is severe. Check probe timeouts vs observed startup/health latency. | | **Liveness probe misconfiguration** | Restarts without resource pressure; `initialDelaySeconds` `progressDeadlineSeconds`; old pods still serving | Check `k8s.deployment.available` vs `.desired`. Admission rejection? Readiness probe failing on new pods? HPA not scaling? | | **Termination signal race** | Brief 5xx bursts correlated with rolling deploys | Endpoint removal races termination. New requests can hit the pod after SIGTERM starts. NGINX gotcha: `STOPSIGNAL SIGTERM` triggers _fast_ shutdown, not graceful — use `STOPSIGNAL SIGQUIT` for graceful drain. Check ingress 502 rate vs rollout timing. | ### Node layer | Mode | Pivotal signal | Investigate | | ----------------------------------- | ----------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | **Node NotReady cascade** | `k8s.node.condition_ready == 0` + mass `Evicted` events | Memory pressure? Disk pressure? Network partition from API server? Inspect kubelet logs, `k8s.node.condition_*` history. | | **Resource eviction** | `status_reason == "Evicted"` + `condition_memory_pressure == 1` on node | Node-level noisy neighbor. QoS order: BestEffort → Burstable → Guaranteed. Identify which pod drove node memory up. | | **Node affinity/selector conflict** | Mass unschedulable pods after label change | K8s events show `FailedScheduling`. Often triggered by cluster upgrades (e.g. `node-role.kubernetes.io/master` → `control-plane`). | ### Control plane | Mode | Pivotal signal | Investigate | | ----------------------------- | ------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------- | | **etcd I/O cascade** | API server latency spike + cluster-wide kubelet heartbeat failures | Disk IOPS, fsync latency (must be 1.0` sustained → CFS throttling. Node has spare CPU; the pod is quota-blocked. - Symptoms of throttling (not the throttle metric itself): liveness probe timeouts, p99 latency 4–16× p50, queue backpressure upstream, Error-reason container terminations. - Average can look healthy while p95 is throttled. Do not trust average alone. ### Restart patterns - `restarts > 0` recently → workload has been restarting. Don't read magnitude into the count (see _Restart count is boolean_); confirm the pattern from K8s `Killing` / `BackOff` event timestamps in `logs-k8seventsreceiver.otel-*`. - Restarts correlated with memory pressure (`memory_limit_utilization → 1.0`) → OOMKilled path. - Restarts without memory/CPU pressure → probe misconfig, app bug, or startup dependency failure. Pull events for `Unhealthy` and `Killing`. ### Termination reasons - `OOMKilled` → memory … ## Source & license This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code. - **Author:** [elastic](https://github.com/elastic) - **Source:** [elastic/agent-skills](https://github.com/elastic/agent-skills) - **License:** Apache-2.0 Install and usage instructions live in the source repository linked above. ## Pricing - **Free** — Free ## Security capabilities Automated source analysis of v0.1.0 — what this tool can access: - **Network access:** no - **Filesystem access:** no - **Shell / process execution:** no - **Environment & secrets:** no - **Dynamic code execution:** no *"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.* ## Versions - **0.1.0** — security scan: passed — Imported from the upstream source. ## Links - Listing page: https://agentstack.voostack.com/l/skill-elastic-agent-skills-k8s-investigation - Seller: https://agentstack.voostack.com/s/elastic - Browse the marketplace: https://agentstack.voostack.com/browse --- Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.