K8s Network Troubleshooting

Kubernetes Network Troubleshooting

Diagnose network connectivity issues within and outside a Kubernetes cluster. Covers pod-to-pod, pod-to-service, ingress, egress, network policies, and Istio service mesh.

Keywords

network, connectivity, connection refused, timeout, unreachable, curl, wget, pod-to-pod, pod-to-service, service mesh, istio, envoy, sidecar, mTLS, network policy, ingress, egress, ClusterIP, NodePort, LoadBalancer, CNI, kube-proxy, iptables, calico, cilium, VirtualService, DestinationRule, Gateway

When to Use This Skill

• Pods cannot reach other pods or services (connection refused / timeout)
• Connection timeouts between services that were previously working
• Ingress controller is not routing external traffic to services
• Egress to external APIs or the internet is blocked
• Intermittent connectivity failures between services
• Istio sidecar injection issues or mTLS failures
• Traffic routing behaves differently than VirtualService rules specify
• Services are reachable by ClusterIP but not by name (check DNS first)
• NodePort or LoadBalancer services are not externally accessible

When NOT to Use

• DNS resolution failures (pods can't resolve names) → use [k8s-dns-troubleshooting](../k8s-dns-troubleshooting)
• Pod crashes, scheduling failures, or OOMKills → use [k8s-namespace-troubleshooting](../k8s-namespace-troubleshooting)
• TLS certificate issuance or renewal failures → use [cert-manager-troubleshooting](../cert-manager-troubleshooting)

Note: If connections fail with "could not resolve host", the problem is DNS, not network. Start with [k8s-dns-troubleshooting](../k8s-dns-troubleshooting). If DNS resolves but the connection still fails, return here.

Related Skills

• [k8s-dns-troubleshooting](../k8s-dns-troubleshooting) - DNS resolution and CoreDNS issues
• [k8s-namespace-troubleshooting](../k8s-namespace-troubleshooting) - General namespace diagnosis
• [k8s-platform-operations](../k8s-platform-operations) - Cluster-wide health checks
• [k8s-security-hardening](../k8s-security-hardening) - Network policies and security controls
• [Shared: Network Policies](../_shared/references/network-policies.md)

Quick Reference

| Task | Command | |------|---------| | Test connectivity from a pod | kubectl exec -n ${NS} ${POD} -- wget -qO- --timeout=5 http://${TARGET}:${PORT} 2>&1 | | Check service endpoints | kubectl get endpoints ${SVC} -n ${NS} | | List network policies | kubectl get networkpolicies -n ${NS} | | Describe a service | kubectl describe svc ${SVC} -n ${NS} | | Check pod IP | kubectl get pod ${POD} -n ${NS} -o jsonpath='{.status.podIP}' | | Check Istio sidecar status | kubectl get pod ${POD} -n ${NS} -o jsonpath='{.spec.containers[*].name}' | | Envoy proxy logs | kubectl logs ${POD} -n ${NS} -c istio-proxy --tail=100 | | Ingress resources | kubectl get ingress -A | | kube-proxy pods | kubectl get pods -n kube-system -l k8s-app=kube-proxy | | CNI pods | kubectl get pods -A -l k8s-app=calico-node or kubectl get pods -A -l k8s-app=cilium |

---

Quick Network Health Check

Run this checklist first. Complete all checks before drawing conclusions.

Step 1: Pod-to-Pod (Same Namespace)

# Find two running pods in the same namespace
kubectl get pods -n ${NS} --field-selector=status.phase=Running -o wide

# Get the IP of the target pod
TARGET_IP=$(kubectl get pod ${TARGET_POD} -n ${NS} -o jsonpath='{.status.podIP}')

# Test connectivity from source pod to target pod IP
kubectl exec -n ${NS} ${SOURCE_POD} -- wget -qO- --timeout=5 http://${TARGET_IP}:${PORT} 2>&1

Step 2: Pod-to-Service (Same Namespace)

# Verify the service exists and has a ClusterIP
kubectl get svc ${SVC} -n ${NS}

# Verify endpoints are populated
kubectl get endpoints ${SVC} -n ${NS}

# Test connectivity via service name
kubectl exec -n ${NS} ${POD} -- wget -qO- --timeout=5 http://${SVC}:${SVC_PORT} 2>&1

Step 3: Pod-to-Service (Cross Namespace)

# Test connectivity to a service in a different namespace
kubectl exec -n ${NS} ${POD} -- wget -qO- --timeout=5 http://${SVC}.${TARGET_NS}.svc.cluster.local:${SVC_PORT} 2>&1

Step 4: Pod-to-External

Requires internet access from the cluster. Substitute any reachable external endpoint in air-gapped or restricted environments.

# Test internet connectivity
kubectl exec -n ${NS} ${POD} -- wget -qO- --timeout=5 http://httpbin.org/get 2>&1 | head -5

# Test specific external endpoint
kubectl exec -n ${NS} ${POD} -- wget -qO- --timeout=5 https://${EXTERNAL_HOST} 2>&1 | head -3

Step 5: Network Policies

# Check for network policies in the namespace
kubectl get networkpolicies -n ${NS}

# Check for policies that restrict egress or ingress
kubectl get networkpolicies -n ${NS} -o json | \
  jq -r '.items[] | "\(.metadata.name)\tTypes: \(.spec.policyTypes // ["Ingress"] | join(", "))"'

Step 6: Istio Sidecar (If Applicable)

# Check if the pod has an istio-proxy sidecar container
kubectl get pod ${POD} -n ${NS} -o jsonpath='{.spec.containers[*].name}' | tr ' ' '\n' | grep istio-proxy

# Check if the namespace has sidecar injection enabled
kubectl get namespace ${NS} -o jsonpath='{.metadata.labels.istio-injection}'

Health Summary Template

Present results using this format:

## Network Health Summary

| Check | Status | Detail |
|-------|--------|--------|
| Pod-to-pod (same NS) | PASS/FAIL/SKIP | Connected / Timeout / Refused / Not tested |
| Pod-to-service (same NS) | PASS/FAIL/SKIP | Endpoints: X, Response: OK / Timeout / Refused |
| Pod-to-service (cross NS) | PASS/FAIL/SKIP | Connected / Blocked / Not tested |
| Pod-to-external | PASS/FAIL/SKIP | Connected / Timeout / Blocked |
| Network policies | NONE/PRESENT | X policies found (Y with egress rules) |
| Istio sidecar injected | YES/NO/N/A | istio-proxy present / absent / Istio not installed |

**Overall: HEALTHY / DEGRADED / UNHEALTHY**

• HEALTHY — All connectivity checks pass
• DEGRADED — Some paths work, others fail (targeted issue)
• UNHEALTHY — Broad connectivity failure

---

Diagnostic Workflow

Use this decision tree after the health check identifies failures.

Connection failing?
├─ Same namespace pod-to-pod fails?
│   ├─ Both pods on same node → CNI issue (Section 7)
│   ├─ Pods on different nodes → CNI cross-node or overlay issue (Section 7)
│   └─ Network policy blocking → Ingress policy on target (Section 5)
├─ Service connection fails but pod IP works?
│   ├─ Endpoints empty → No ready pods backing the service (Section 2)
│   ├─ Port mismatch → Service port vs targetPort vs containerPort (Section 2)
│   ├─ kube-proxy not running → No iptables/IPVS rules for ClusterIP (Section 7)
│   └─ Istio routing override → VirtualService or DestinationRule (Section 6)
├─ Cross-namespace fails but same-namespace works?
│   ├─ Network policy restricting cross-NS traffic → Egress or ingress rules (Section 5)
│   ├─ Istio mTLS mode mismatch → STRICT on one side, no sidecar on other (Section 6)
│   └─ DNS issue → Resolve ${SVC}.${NS}.svc.cluster.local (→ k8s-dns-troubleshooting)
├─ External/egress fails?
│   ├─ All pods affected → Cluster-wide egress blocked (Section 3)
│   ├─ Only some pods → Egress network policy (Section 5)
│   ├─ Istio sidecar blocking → Outbound traffic policy (Section 6)
│   └─ NAT/SNAT issue → Node cannot route to internet (Section 3)
├─ Ingress fails?
│   ├─ Ingress controller not running → Controller pods down (Section 4)
│   ├─ Ingress resource misconfigured → Backend service or path (Section 4)
│   ├─ TLS termination failing → Certificate or secret issue (Section 4)
│   └─ Istio Gateway misconfigured → Gateway or VirtualService (Section 6)
└─ Intermittent failures?
    ├─ Some requests succeed → Load balancing to unhealthy pod (Section 2)
    ├─ Timeouts then recovery → Conntrack table full or CNI flap (Section 7)
    └─ Istio circuit breaking → DestinationRule outlier detection (Section 6)

---

Section 1: Pod-to-Pod Connectivity

Direct pod-to-pod communication uses pod IPs, bypassing services and kube-proxy.

Verifying Pod IPs and Placement

# Get pod IPs and node placement
kubectl get pods -n ${NS} -o wide

# Confirm pods have IPs assigned (no IP = CNI problem)
kubectl get pod ${POD} -n ${NS} -o jsonpath='{.status.podIP}'

# Check if pods are on the same or different nodes
kubectl get pod ${SOURCE_POD} -n ${NS} -o jsonpath='{.spec.nodeName}'
kubectl get pod ${TARGET_POD} -n ${NS} -o jsonpath='{.spec.nodeName}'

Testing Pod-to-Pod

# Direct connectivity test using pod IP
TARGET_IP=$(kubectl get pod ${TARGET_POD} -n ${NS} -o jsonpath='{.status.podIP}')
kubectl exec -n ${NS} ${SOURCE_POD} -- wget -qO- --timeout=5 http://${TARGET_IP}:${PORT} 2>&1

# If wget is unavailable, test with /dev/tcp (bash-based images)
kubectl exec -n ${NS} ${SOURCE_POD} -- bash -c "echo > /dev/tcp/${TARGET_IP}/${PORT} && echo OPEN || echo CLOSED" 2>&1

# Check if the target pod is listening on the expected port
kubectl exec -n ${NS} ${TARGET_POD} -- sh -c 'netstat -tlnp 2>/dev/null || ss -tlnp 2>/dev/null' | head -20

Same Node vs Cross Node

# If same-node pod-to-pod fails: CNI plugin local routing is broken
# If cross-node works: unlikely to be CNI, check network policy on target

# If cross-node pod-to-pod fails: overlay network issue
# Check CNI pods are running on both nodes (use the label for the installed CNI)
kubectl get pods -n kube-system -l k8s-app=calico-node -o wide --field-selector spec.nodeName=${NODE1} 2>/dev/null
kubectl get pods -n kube-system -l k8s-app=cilium -o wide --field-selector spec.nodeName=${NODE1} 2>/dev/null

Interpreting Failures

| Symptom | Likely Cause | |---------|-------------| | Connection refused | Target pod is not listening on that port | | Connection timed out | Network path blocked (policy, CNI, or routing) | | No pod IP assigned | CNI plugin failed to allocate an IP | | Works same-node, fails cross-node | Overlay/tunnel issue between nodes |

---

Section 2: Pod-to-Service Connectivity

Services route traffic through ClusterIP (kube-proxy rules) to pod endpoints.

Checking Service Configuration

# Service details: ClusterIP, ports, selector
kubectl describe svc ${SVC} -n ${NS}

# Verify selector matches pod labels
SVC_SELECTOR=$(kubectl get svc ${SVC} -n ${NS} -o jsonpath='{.spec.selector}')
echo "Service selector: ${SVC_SELECTOR}"
kubectl get pods -n ${NS} -l "${LABEL_KEY}=${LABEL_VALUE}" -o wide

# Port mapping — the three ports that must align
kubectl get svc ${SVC} -n ${NS} -o jsonpath='{range .spec.ports[*]}port:{.port} targetPort:{.targetPort} protocol:{.protocol}{"\n"}{end}'

The Three-Port Model

Client → Service:port → Pod:targetPort → Container listens on containerPort

- service.spec.ports[].port      = port clients connect to
- service.spec.ports[].targetPort = port on the pod (defaults to .port if omitted)
- container.ports[].containerPort = port the app listens on (must match targetPort)

A mismatch between targetPort and the actual port the container listens on is a common cause of "connection refused" after DNS resolves.

Checking Endpoints

# Endpoints show which pod IPs back the service
kubectl get endpoints ${SVC} -n ${NS}

# Detailed endpoint view
kubectl get endpoints ${SVC} -n ${NS} -o yaml

# If using EndpointSlices (Kubernetes 1.21+)
kubectl get endpointslices -n ${NS} -l kubernetes.io/service-name=${SVC}

Empty Endpoints Causes

| Cause | Diagnostic | |-------|-----------| | No pods match selector | Compare svc.spec.selector with pod.metadata.labels | | Pods not ready | Check kubectl get pods -n ${NS} — pods must pass readiness probes | | Wrong namespace | Service and pods must be in the same namespace (for selector matching) | | Readiness probe failing | kubectl describe pod ${POD} -n ${NS} — check probe events |

Service Types

# ClusterIP (default) — internal only
kubectl get svc ${SVC} -n ${NS} -o jsonpath='{.spec.type}'

# NodePort — verify port allocation
kubectl get svc ${SVC} -n ${NS} -o jsonpath='{range .spec.ports[*]}nodePort:{.nodePort}{"\n"}{end}'

# LoadBalancer — check external IP assignment
kubectl get svc ${SVC} -n ${NS} -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

# If LoadBalancer shows , the cloud controller hasn't provisioned the LB
kubectl describe svc ${SVC} -n ${NS} | grep -A5 Events

Testing Service Connectivity

# Via service name (tests DNS + kube-proxy + pod)
kubectl exec -n ${NS} ${POD} -- wget -qO- --timeout=5 http://${SVC}:${SVC_PORT} 2>&1

# Via ClusterIP directly (bypasses DNS, tests kube-proxy + pod)
CLUSTER_IP=$(kubectl get svc ${SVC} -n ${NS} -o jsonpath='{.spec.clusterIP}')
kubectl exec -n ${NS} ${POD} -- wget -qO- --timeout=5 http://${CLUSTER_IP}:${SVC_PORT} 2>&1

# If service works by ClusterIP but not by name → DNS problem
# If service works by pod IP but not by ClusterIP → kube-proxy problem
# If neither works → pod/network problem

---

Section 3: Egress / External Connectivity

Pods reaching endpoints outside the cluster.

Testing External Connectivity

These examples use httpbin.org as a public test endpoint. Substitute any reachable external host in air-gapped or restricted environments.

# Test general internet access
kubectl exec -n ${NS} ${POD} -- wget -qO- --timeout=5 http://httpbin.org/get 2>&1 | head -5

# Test specific external API
kubectl exec -n ${NS} ${POD} -- wget -qO- --timeout=5 https://${EXTERNAL_HOST}${PATH} 2>&1 | head -5

# If wget/curl unavailable, test TCP connectivity
kubectl exec -n ${NS} ${POD} -- sh -c "echo | nc -w5 ${EXTERNAL_HOST} ${PORT}" 2>&1

Common Egress Failures

| Symptom | Likely Cause | Diagnostic | |---------|-------------|-----------| | All pods can't reach external | Cluster-wide egress issue | Check node internet access, NAT gateway | | Only some pods blocked | Egress network policy | kubectl get networkpolicies -n ${NS} | | HTTPS fails, HTTP works | TLS interception or proxy | Check for corporate proxy env vars | | Timeout to specific host | Firewall or security group | Check cloud provider firewall rules | | Connection refused to external | Target service rejecting | Verify external service is up |

NAT and SNAT

# Pods typically use the node's IP for outbound traffic (SNAT)
# Verify the node has internet access

# Check if a pod sees the expected source IP
kubectl exec -n ${NS} ${POD} -- wget -qO- --timeout=5 http://httpbin.org/ip 2>&1

# If pods use an egress gateway (Istio or CNI-specific), check its health
kubectl get pods -n istio-system -l istio=egressgateway 2>/dev/null

Proxy Configuration

# Check if pods expect a proxy
kubectl exec -n ${NS} ${POD} -- sh -c 'echo "HTTP_PROXY=$HTTP_PROXY HTTPS_PROXY=$HTTPS_PROXY NO_PROXY=$NO_PROXY"'

# Check if cluster-internal traffic is excluded from proxy
# NO_PROXY should include: .svc,.svc.cluster.local,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16

---

Section 4: Ingress Connectivity

External traffic reaching services through an ingress controller.

Ingress Controller Health

# Find the ingress controller pods (common labels)
kubectl get pods -A -l app.kubernetes.io/name=ingress-nginx -o wide
kubectl get pods -A -l app=istio-ingressgateway -o wide
kubectl get pods -A -l app.kubernetes.io/name=traefik -o wide

# Check controller service (must have an external IP or be NodePort)
kubectl get svc -A | grep -E 'ingress|gateway'

# Controller logs
kubectl logs -n ${INGRESS_NS} -l app.kubernetes.io/name=ingress-nginx --tail=100

Ingress Resource Configuration

# List all ingress resources
kubectl get ingress -A

# Describe a specific ingress
kubectl describe ingress ${INGRESS_NAME} -n ${NS}

# Check the backend service references
kubectl get ingress ${INGRESS_NAME} -n ${NS} -o jsonpath='{range .spec

…

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [foxj77](https://github.com/foxj77)
- **Source:** [foxj77/claude-code-skills](https://github.com/foxj77/claude-code-skills)
- **License:** MIT

Install and usage instructions live in the source repository linked above.