Observability K8s Investigation

Kubernetes Investigation

Diagnose Kubernetes issues using OTel telemetry collected via EDOT (Elastic Distribution of OpenTelemetry) and the kube-stack collector. Correlate cluster state, pod runtime metrics, K8s events, application logs, and APM to identify root cause across the workload, node, and control-plane layers.

Scope

In scope: OTel-receiver-namespaced indices (metrics-kubeletstatsreceiver.otel-*, metrics-k8sclusterreceiver.otel-*, logs-k8seventsreceiver.otel-*, logs-k8sobjectsreceiver.otel-*) and OTel semantic conventions (k8s.pod.name, k8s.namespace.name, k8s.container.restarts).

Out of scope:

• The legacy Elastic Agent Kubernetes integration (metrics-kubernetes.*, logs-kubernetes.*, kubernetes.* fields).

Being deprecated — do not author queries against these paths.

• APM-layer analysis (service SLO breaches, transaction error rates, upstream dependency health). Different domain —

once a K8s root cause is ruled in or out, APM investigation continues outside this skill.

• Cluster provisioning, capacity planning, cost optimization. Different domain.

Guidelines

These apply to every investigation. When in doubt, re-read them before writing the synthesis.

Absence of evidence is not evidence. Do not confabulate from empty results. If log queries return 0 rows, logs are likely not collected or the pod has no recent lines — this does not mean "dependency unavailable" or any other specific failure mode. Report no_logs_available and weight remaining signals accordingly.

Empty dependency data ≠ upstream healthy. Services without APM instrumentation (load generators, workers) emit no destination metrics. Report insufficient_dependency_data, not "upstreams OK."

Co-symptoms are not causes. Two services degrading simultaneously usually share an upstream, not a causal link. Only attribute causation when (a) one service's degradation clearly precedes the other's, and (b) the delta is large (>5× error rate, >3× latency).

OOMKilled ≠ memory leak by default. The limit may simply be undersized for the workload's working set. Compare against a 7-day baseline at the same hour-of-day before claiming a leak.

Error-termination ≠ application bug by default. Check k8s.pod.cpu_limit_utilization first. CFS throttling driving liveness probe timeouts is the most common misdiagnosis in this space.

Average CPU hides throttling. A pod can look healthy at 40–60% average cpu_limit_utilization while being throttled severely at p99. Linux enforces CPU limits in 100ms periods; bursty workloads hit quota mid-period and stall. Look at max and p95, not just average.

Restart count is boolean, not a counter. k8s.container.restarts is pulled directly from the K8s API and may be pruned by the kubelet at any time, so the absolute value is unreliable. Treat it as == 0 (no recent restarts) vs > 0 (recently restarting); do not derive backoff timing or "linear vs exponential" patterns from it. Confirm the restart pattern via K8s Killing / BackOff events instead.

Prefer to report uncertainty over manufacturing confidence. If the evidence is ambiguous, the synthesis should say so. Competing hypotheses are a valid output.

Indices and fields

Where to look

| Signal | Index pattern | Use | | --------------------- | --------------------------------------------------- | ------------------------------------------------------------------- | | Pod/container runtime | metrics-kubeletstatsreceiver.otel-* | CPU, memory, network, filesystem. Utilization ratios. | | Cluster state | metrics-k8sclusterreceiver.otel-* | Restarts, phase, last-terminated reason, HPA, quota, node condition | | K8s events | logs-k8seventsreceiver.otel-* | Killing, BackOff, FailedScheduling, Evicted, image pull events | | K8s object snapshots | logs-k8sobjectsreceiver.otel-* | Deployment/service/configmap state over time | | Application logs | logs-*.otel-* | body.text, severity_text, filtered by k8s.pod.name | | APM | traces-*.otel-*, metrics-service_*.otel-default | Correlate via service.name + K8s resource attrs | | ML anomalies | .ml-anomalies-* | Memory-growth, restart-rate, throttle jobs (if configured) |

Key fields

Flat OTel paths work in ES|QL. Prefer the flat form for readability; the nested resource.attributes.* form is for raw log documents only.

| Field | Index | What it is | | ------------------------------------------------ | --------------------------- | ------------------------------------------------------- | | k8s.pod.name | all k8s | Pod name | | k8s.namespace.name | all k8s | Namespace | | k8s.container.name | all k8s | Container within pod | | k8s.deployment.name | k8sclusterreceiver + others | Parent deployment | | k8s.pod.phase | k8sclusterreceiver | Pending=1/Running=2/Succeeded=3/Failed=4/Unknown=5 | | k8s.container.restarts | k8sclusterreceiver | Total container restart count | | k8s.container.status.last_terminated_reason | k8sclusterreceiver | OOMKilled, Error, Completed, ContainerCannotRun | | k8s.pod.status_reason | k8sclusterreceiver | Pod-level reason (Evicted, NodeLost) | | k8s.pod.memory_limit_utilization | kubeletstatsreceiver | 0.0–1.0+ (can exceed 1 transiently before OOM) | | k8s.pod.cpu_limit_utilization | kubeletstatsreceiver | 0.0–N (frequently >1 under CFS throttling) | | k8s.pod.memory.usage / .working_set | kubeletstatsreceiver | Bytes | | k8s.node.condition_memory_pressure | k8sclusterreceiver | 1 = pressure, 0 = ok | | k8s.node.condition_ready | k8sclusterreceiver | 0 = NotReady | | k8s.hpa.current_replicas / .desired_replicas | k8sclusterreceiver | HPA state | | attributes.k8s.event.reason | k8seventsreceiver | Event reason (filter on this) | | body.text | k8seventsreceiver / logs | Event message / log message | | k8s.object.name | k8seventsreceiver | involvedObject name (log attribute, use flat form) |

Field availability

Several fields above are off by default in stock kube-stack collectors and require explicit configuration. Verify presence before relying on them; if absent, fall back as noted and call out the substitution in the synthesis.

| Field | Why it might be missing | Fall-back | | ------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | | k8s.container.status.last_terminated_reason | Optional metric in k8sclusterreceiver; gated behind metrics_collected.metadata config. | Infer from K8s Killing / OOMKilling events in logs-k8seventsreceiver.otel-* and exit codes in app logs. | | k8s.pod.status_reason | Same — optional metric on k8sclusterreceiver. | Infer from events: Evicted, NodeLost, Preempted. | | k8s.pod.cpu_limit_utilization / memory_limit_utilization | Only emitted when the pod has the corresponding limit set, and the kubeletstatsreceiver metric is enabled. | Compute manually as k8s.pod.cpu.usage / from k8sclusterreceiver, or use absolute usage trending against a baseline. | | k8s.node.condition_memory_pressure | Gated behind k8sclusterreceiver node_conditions_to_report (default omits this). | Compare k8s.node.memory.usage against k8s.node.allocatable_memory, or look for Evicted events on the node. |

If a fall-back is used, note it in the synthesis (e.g. (via memory.usage; limit_utilization not collected)) so the reader knows the signal is indirect.

ES|QL gotchas

Before writing queries, know these. Each of them silently produces wrong answers rather than failing loudly.

VALUES() returns scalar for single distinct value, array for multiple. Templating that assumes array shape (e.g. | first) extracts the first character of the string when scalar. Use MV_FIRST(VALUES(...)) or handle both.

PERCENTILE does not work on OTel histogram type (as of 8.15). For APM duration percentiles, use AVG on the aggregate_metric_double summary field (AVG(transaction.duration.summary) divides sum by value_count). For true percentiles, fall back to Kibana Query DSL.

COUNT(agg_metric_double) returns value_count (events), not doc count. SUM(field) gives the sum component; AVG(field) gives sum/value_count. Do not use SUM(transaction.duration.summary) as an event-count proxy — it returns total duration.

K8s metrics use flat OTel field paths in ES|QL. k8s.pod.name, not resource.attributes.k8s.pod.name. The nested form is for raw log documents.

Failure-mode taxonomy

Vocabulary for classification, not a decision tree. Use the pivotal-signal column to recognize which mode you're looking at; use "Investigate" to know what else should corroborate.

Workload layer

| Mode | Pivotal signal | Investigate | | ----------------------------------- | -------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | OOMKilled | last_terminated_reason == "OOMKilled" + memory_limit_utilization → 1.0 | Monotonic rise (leak) vs. load-driven spike? Compare current trend to 7-day baseline. Check heap metrics (JVM, Go, Node) for GC pressure. | | CPU throttling → Error exit | cpu_limit_utilization > 1.0 + last_terminated_reason == "Error" | Liveness/readiness probe timeouts from CFS throttling. Average CPU can look fine (40–60%) while p99 throttle is severe. Check probe timeouts vs observed startup/health latency. | | Liveness probe misconfiguration | Restarts without resource pressure; initialDelaySeconds progressDeadlineSeconds; old pods still serving | Check k8s.deployment.available vs .desired. Admission rejection? Readiness probe failing on new pods? HPA not scaling? | | Termination signal race | Brief 5xx bursts correlated with rolling deploys | Endpoint removal races termination. New requests can hit the pod after SIGTERM starts. NGINX gotcha: STOPSIGNAL SIGTERM triggers fast shutdown, not graceful — use STOPSIGNAL SIGQUIT for graceful drain. Check ingress 502 rate vs rollout timing. |

Node layer

| Mode | Pivotal signal | Investigate | | ----------------------------------- | ----------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | Node NotReady cascade | k8s.node.condition_ready == 0 + mass Evicted events | Memory pressure? Disk pressure? Network partition from API server? Inspect kubelet logs, k8s.node.condition_* history. | | Resource eviction | status_reason == "Evicted" + condition_memory_pressure == 1 on node | Node-level noisy neighbor. QoS order: BestEffort → Burstable → Guaranteed. Identify which pod drove node memory up. | | Node affinity/selector conflict | Mass unschedulable pods after label change | K8s events show FailedScheduling. Often triggered by cluster upgrades (e.g. node-role.kubernetes.io/master → control-plane). |

Control plane

| Mode | Pivotal signal | Investigate | | ----------------------------- | ------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------- | | etcd I/O cascade | API server latency spike + cluster-wide kubelet heartbeat failures | Disk IOPS, fsync latency (must be 1.0` sustained → CFS throttling. Node has spare CPU; the pod is quota-blocked.

• Symptoms of throttling (not the throttle metric itself): liveness probe timeouts, p99 latency 4–16× p50, queue

backpressure upstream, Error-reason container terminations.

• Average can look healthy while p95 is throttled. Do not trust average alone.

Restart patterns

• restarts > 0 recently → workload has been restarting. Don't read magnitude into the count (see _Restart count is

boolean_); confirm the pattern from K8s Killing / BackOff event timestamps in logs-k8seventsreceiver.otel-*.

• Restarts correlated with memory pressure (memory_limit_utilization → 1.0) → OOMKilled path.
• Restarts without memory/CPU pressure → probe misconfig, app bug, or startup dependency failure. Pull events for

Unhealthy and Killing.

Termination reasons

• OOMKilled → memory

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: elastic
• Source: elastic/agent-skills
• License: Apache-2.0

Install and usage instructions live in the source repository linked above.