Pod Troubleshooting

Skill: Pod Troubleshooting

> Expertise: Systematic K8s failure diagnosis — from symptom to root cause in under 10 commands.

When to load

When a pod is not Running, a service is unreachable, or a deployment is stuck.

Diagnostic Decision Tree

Pod not Running?
├── Status: Pending
│   ├── No nodes match → check node selectors, taints, resource requests
│   └── PVC not bound → check StorageClass, PV availability
├── Status: CrashLoopBackOff
│   ├── Exit code 0 → process exited cleanly but K8s restarts it → check command
│   ├── Exit code 1 → app error → check logs
│   ├── Exit code 137 → OOMKilled → increase memory limit
│   └── Exit code 143 → SIGTERM not handled → fix graceful shutdown
├── Status: ImagePullBackOff
│   ├── Image doesn't exist → check tag/digest
│   └── Registry auth fails → check imagePullSecret
└── Status: Error / Init:Error
    └── Init container failed → check init container logs

Command Cheatsheet

# 1. Overview — what's wrong
kubectl get pods -n  -o wide
kubectl describe pod  -n           # events section is the first place to look

# 2. Logs
kubectl logs  -n                   # current container
kubectl logs  -n  --previous       # last crashed container (CrashLoop)
kubectl logs  -n  -c    # specific container in multi-container pod

# 3. Exec into running pod
kubectl exec -it  -n  -- /bin/sh

# 4. Resource pressure check
kubectl top nodes
kubectl top pods -n 

# 5. Events (cluster-wide, sorted)
kubectl get events -n  --sort-by='.lastTimestamp' | tail -20

# 6. Debug ephemeral container (no exec needed — distroless images)
kubectl debug -it  -n  --image=busybox:latest --target=

# 7. Node-level debug
kubectl debug node/ -it --image=ubuntu

CrashLoopBackOff Runbook

# Step 1: Get exit code
kubectl describe pod  -n  | grep -A5 "Last State:"

# Step 2: Get crash logs (may only appear in --previous)
kubectl logs  -n  --previous --tail=100

# Step 3: Check if OOMKilled
kubectl describe pod  -n  | grep -i "OOMKilled\|Reason:"
# If OOMKilled → increase memory limit or find memory leak

# Step 4: Check security context (common in restricted namespaces)
# Error: "permission denied" or "operation not permitted" → readOnlyRootFilesystem or dropped capabilities

Pending Pod Runbook

# Check why pod can't be scheduled
kubectl describe pod  -n  | grep -A20 "Events:"

# Common causes:
# "Insufficient cpu/memory" → check node capacity and pod requests
kubectl describe nodes | grep -A5 "Allocated resources:"

# "node(s) had taints that the pod didn't tolerate"
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

# "0/3 nodes are available: 3 node(s) didn't match node affinity"
# → check pod nodeSelector / affinity vs node labels
kubectl get nodes --show-labels

Service Connectivity Runbook

# Is the service selecting the right pods?
kubectl get endpoints  -n        # should show pod IPs; empty = selector mismatch

# Test DNS resolution from within cluster
kubectl run -it --rm debug --image=busybox --restart=Never -- nslookup ..svc.cluster.local

# Test HTTP connectivity
kubectl run -it --rm debug --image=curlimages/curl --restart=Never -- curl -v http://..svc.cluster.local:/health

# Check NetworkPolicy blocking traffic
# Install Hubble CLI (Cilium) or use:
kubectl exec -n kube-system  -- cilium monitor --from-pod 

OOMKilled Prevention

# Find actual peak memory usage via metrics
kubectl top pods -n  --sort-by=memory

# Check Vertical Pod Autoscaler recommendation (if VPA installed)
kubectl describe vpa  -n  | grep -A10 "Recommendation:"

# Rule of thumb: limit = 1.5× observed peak; request = 0.6× observed peak

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: sawrus
• Source: sawrus/agent-guides
• License: MIT
• Homepage: https://sawrus.github.io/agent-guides

Install and usage instructions live in the source repository linked above.