K8s Dns Troubleshooting

Kubernetes DNS Troubleshooting

Investigate and diagnose internal Kubernetes DNS (CoreDNS). Perform a quick health assessment to determine whether cluster DNS is functioning correctly, then deep-dive into root causes when issues are found.

Keywords

coredns, kube-dns, dns, resolve, nslookup, dig, service discovery, NXDOMAIN, dns timeout, dns lookup, ndots, search domain, cluster.local, Corefile, dns policy, dns resolution, dns health, dns check, name resolution, dns-default, cluster-first

When to Use This Skill

• Pods cannot resolve service names (NXDOMAIN or timeouts)
• CoreDNS pods are crashing, restarting, or not running
• DNS lookups are slow (high latency on service calls)
• Service discovery is broken after a cluster upgrade or config change
• A general DNS health check is needed to confirm the cluster is healthy
• Applications log connection errors that trace back to DNS failures
• Cross-namespace service resolution is failing
• Headless service DNS records are not returning pod IPs

When NOT to Use

• DNS records for external providers are not syncing → use [external-dns-troubleshooting](../external-dns-troubleshooting)
• TLS certificate issuance failures → use [cert-manager-troubleshooting](../cert-manager-troubleshooting)
• Service exists but has no endpoints → use [k8s-namespace-troubleshooting](../k8s-namespace-troubleshooting)

Note: Pods failing to resolve external domains (e.g., api.example.com) are in scope — this is an upstream forwarding issue diagnosed in Section 3 (Corefile) and Section 5 (network connectivity). ExternalDNS is a separate concern about publishing records to providers.

Related Skills

• [external-dns-troubleshooting](../external-dns-troubleshooting) - External DNS record management
• [k8s-namespace-troubleshooting](../k8s-namespace-troubleshooting) - General namespace diagnosis
• [k8s-platform-operations](../k8s-platform-operations) - Cluster-wide health checks
• [k8s-network-troubleshooting](../k8s-network-troubleshooting) - Network connectivity and service mesh issues
• [k8s-security-hardening](../k8s-security-hardening) - Network policies that may block DNS
• [Shared: Network Policies](../_shared/references/network-policies.md)

Quick Reference

| Task | Command | |------|---------| | CoreDNS pod status | kubectl get pods -n kube-system -l k8s-app=kube-dns | | CoreDNS logs | kubectl logs -n kube-system -l k8s-app=kube-dns --tail=100 | | DNS service ClusterIP | kubectl get svc kube-dns -n kube-system | | DNS endpoints | kubectl get endpoints kube-dns -n kube-system | | Test DNS from existing pod | kubectl exec -n ${NS} ${POD} -- nslookup kubernetes.default | | Find a running pod to exec into | kubectl get pods -A --field-selector=status.phase=Running -o jsonpath='{.items[0].metadata.namespace} {.items[0].metadata.name}' | | View Corefile | kubectl get configmap coredns -n kube-system -o yaml | | Pod DNS config | kubectl exec ${POD} -n ${NS} -- cat /etc/resolv.conf |

---

Quick DNS Health Check

Run this checklist first for a fast pass/fail indicator of cluster DNS health. Complete all checks before drawing conclusions.

Step 1: CoreDNS Pods

# Are CoreDNS pods running?
kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide

# Check for restarts (indicates instability)
kubectl get pods -n kube-system -l k8s-app=kube-dns -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{range .status.containerStatuses[*]}{.restartCount}{end}{"\n"}{end}'

Step 2: DNS Service and Endpoints

# kube-dns Service must exist with a ClusterIP
kubectl get svc kube-dns -n kube-system

# Endpoints must point to CoreDNS pod IPs
kubectl get endpoints kube-dns -n kube-system

Step 3: Resolution Test

Test DNS from an existing running pod. No new workloads are deployed.

# Find a running pod to use as a test subject
kubectl get pods -A --field-selector=status.phase=Running -o jsonpath='{range .items[0]}{.metadata.namespace}{" "}{.metadata.name}{"\n"}{end}'

# Verify the pod's resolv.conf points to the kube-dns ClusterIP
kubectl exec -n ${NS} ${POD} -- cat /etc/resolv.conf

# Test resolution (nslookup available in most images)
kubectl exec -n ${NS} ${POD} -- nslookup kubernetes.default.svc.cluster.local

# If nslookup is not available, try wget or getent as fallbacks
kubectl exec -n ${NS} ${POD} -- getent hosts kubernetes.default.svc.cluster.local
kubectl exec -n ${NS} ${POD} -- wget -qO- --timeout=5 http://kubernetes.default.svc:443 2>&1 | head -1

If nslookup/getent are not available in the pod image, rely on indirect indicators: Steps 1-2 passing (CoreDNS pods running, service and endpoints healthy) plus no DNS errors in CoreDNS logs (Section 6) is strong evidence DNS is functional.

If resolution fails, see Section 2 for comprehensive testing. If it passes, see Section 2 for deeper checks (cross-namespace, headless, external resolution).

Health Summary Template

Present results using this format:

## DNS Health Summary

| Check | Status | Detail |
|-------|--------|--------|
| CoreDNS pods running | PASS/FAIL | X/Y pods ready, Z restarts |
| kube-dns Service exists | PASS/FAIL | ClusterIP: X.X.X.X |
| kube-dns Endpoints populated | PASS/FAIL | X endpoints |
| Resolve kubernetes.default (exec) | PASS/FAIL/SKIP | Resolved to X.X.X.X / Timeout / NXDOMAIN / No tools available |
| CoreDNS health endpoint | PASS/FAIL | localhost:8080/health returns OK |
| CoreDNS restart count | PASS/WARN | Total restarts across pods |

**Overall: HEALTHY / DEGRADED / UNHEALTHY**

• HEALTHY — All checks pass, zero restarts
• DEGRADED — Resolution works but CoreDNS has restarts or reduced replicas
• UNHEALTHY — Resolution fails or CoreDNS pods are down

---

Diagnostic Workflow

When the health check reports DEGRADED or UNHEALTHY, use this decision tree to identify root causes.

DNS resolution failing?
├─ CoreDNS pods not running?
│   ├─ Pods Pending → Scheduling issue (Section 1)
│   ├─ Pods CrashLoopBackOff → Bad Corefile or OOM (Section 1, Section 3)
│   └─ Deployment missing → CoreDNS not installed or deleted (Section 1)
├─ CoreDNS pods running but resolution fails?
│   ├─ kube-dns Service missing ClusterIP → Service deleted (Section 1)
│   ├─ kube-dns Endpoints empty → Label mismatch or pods not ready (Section 1)
│   ├─ Pod /etc/resolv.conf wrong → dnsPolicy misconfigured (Section 4)
│   ├─ Internal names fail, external works → Corefile zone config (Section 3)
│   ├─ External names fail, internal works → Upstream forwarder issue (Section 3)
│   ├─ All lookups timeout → Network path blocked (Section 5)
│   └─ Intermittent failures → Overloaded CoreDNS or conntrack (Section 1, Section 5)
├─ DNS slow but eventually resolves?
│   ├─ High ndots causing extra queries → ndots config (Section 4)
│   ├─ CoreDNS CPU/memory saturated → Resource limits (Section 1)
│   └─ Upstream DNS slow → Forwarder latency (Section 3)
└─ Only some pods affected?
    ├─ Check pod dnsPolicy → Per-pod DNS config (Section 4)
    ├─ Network policy blocking egress → UDP/TCP 53 blocked (Section 5)
    └─ Node-specific issue → kube-proxy or iptables (Section 5)

---

Section 1: CoreDNS Component Health

# Deployment status
kubectl get deploy coredns -n kube-system
kubectl describe deploy coredns -n kube-system

# Pod details — check status, restarts, node placement
kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide

# Resource consumption (requires metrics-server)
kubectl top pods -n kube-system -l k8s-app=kube-dns

# Check for OOMKills
kubectl get pods -n kube-system -l k8s-app=kube-dns -o json | \
  jq -r '.items[] | .metadata.name as $pod | .status.containerStatuses[]? | select(.lastState.terminated.reason == "OOMKilled") | "\($pod)\tOOMKilled"'

# Resource requests and limits
kubectl get deploy coredns -n kube-system -o jsonpath='{.spec.template.spec.containers[0].resources}' | jq .

# Events related to CoreDNS
kubectl get events -n kube-system --field-selector involvedObject.name=coredns --sort-by='.lastTimestamp'

# Service and endpoints
kubectl get svc kube-dns -n kube-system -o yaml
kubectl get endpoints kube-dns -n kube-system -o yaml

What to Look For

| Symptom | Likely Cause | Diagnostic | |---------|-------------|-----------| | 0/2 pods ready | CoreDNS crashed or not scheduled | Check events and logs | | High restart count | OOMKill or Corefile error | Check terminated reason and resource limits | | Endpoints empty | Pods not ready or label mismatch | Verify pod labels match k8s-app=kube-dns | | CPU near limits | High query volume | Check if CPU is near limits; recommend increasing CPU limits or adding replicas | | Memory near limits | Large cache or zone data | Check if memory is near limits; recommend increasing memory limits |

---

Section 2: DNS Resolution Testing

Test DNS using existing running pods — no new workloads are deployed. Pick a pod that is already running in the cluster.

Finding a Test Pod

# List running pods (pick one with a shell — alpine, debian, ubuntu images work well)
kubectl get pods -A --field-selector=status.phase=Running -o wide

# Check what tools are available in a pod
kubectl exec -n ${NS} ${POD} -- sh -c 'which nslookup dig getent wget curl 2>/dev/null'

Not all container images include DNS tools. Try commands in this order of preference: nslookup, getent hosts, wget/curl (connection attempt reveals DNS). If no tools are available, use indirect methods (resolv.conf inspection + CoreDNS log analysis).

Basic Resolution Tests

# Resolve the Kubernetes API service (must always work)
kubectl exec -n ${NS} ${POD} -- nslookup kubernetes.default.svc.cluster.local

# Resolve with fully qualified name vs short name
kubectl exec -n ${NS} ${POD} -- sh -c \
  'echo "--- FQDN ---" && nslookup kubernetes.default.svc.cluster.local && echo "--- Short ---" && nslookup kubernetes.default'

# Resolve a service in a specific namespace
kubectl exec -n ${NS} ${POD} -- nslookup ${SERVICE}.${TARGET_NS}.svc.cluster.local

# Test external resolution (upstream forwarding)
kubectl exec -n ${NS} ${POD} -- nslookup www.google.com

Fallback: getent or wget

# If nslookup is not available, use getent
kubectl exec -n ${NS} ${POD} -- getent hosts kubernetes.default.svc.cluster.local
kubectl exec -n ${NS} ${POD} -- getent hosts ${SERVICE}.${TARGET_NS}.svc.cluster.local

# If neither is available, a wget/curl connection attempt tests DNS indirectly
# (DNS failure shows as "bad address" or "could not resolve host")
kubectl exec -n ${NS} ${POD} -- wget -qO- --timeout=3 http://${SERVICE}.${TARGET_NS}.svc.cluster.local 2>&1 | head -3

Targeted Queries Against CoreDNS Directly

# Get kube-dns ClusterIP
DNS_IP=$(kubectl get svc kube-dns -n kube-system -o jsonpath='{.spec.clusterIP}')

# Query CoreDNS directly by specifying the nameserver (bypasses pod resolv.conf)
kubectl exec -n ${NS} ${POD} -- nslookup kubernetes.default.svc.cluster.local ${DNS_IP}

Indirect DNS Verification (No Exec Required)

When kubectl exec is unavailable or pod images lack tools, verify DNS through infrastructure checks:

# 1. Confirm pod resolv.conf nameserver matches kube-dns ClusterIP
DNS_IP=$(kubectl get svc kube-dns -n kube-system -o jsonpath='{.spec.clusterIP}')
echo "kube-dns ClusterIP: ${DNS_IP}"

# 2. Verify CoreDNS is serving queries (check logs for recent activity)
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50 --since=5m

# 3. Check CoreDNS health endpoint from within a CoreDNS pod
COREDNS_POD=$(kubectl get pods -n kube-system -l k8s-app=kube-dns -o jsonpath='{.items[0].metadata.name}')
kubectl exec -n kube-system ${COREDNS_POD} -- wget -qO- http://localhost:8080/health 2>&1
kubectl exec -n kube-system ${COREDNS_POD} -- wget -qO- http://localhost:8181/ready 2>&1

# 4. Check for DNS-related errors in application pod logs
kubectl logs -n ${NS} ${POD} --tail=200 | grep -iE 'dns|resolve|NXDOMAIN|lookup|name.resolution|no.such.host|could.not.resolve'

Headless Service Verification

Headless services (ClusterIP: None) return individual pod IPs instead of a single VIP.

# Verify headless service has endpoints (indirect check — no exec needed)
kubectl get endpoints ${HEADLESS_SVC} -n ${NAMESPACE} -o jsonpath='{.subsets[*].addresses[*].ip}'

# If exec is available, resolve and confirm multiple A records
kubectl exec -n ${NS} ${POD} -- nslookup ${HEADLESS_SVC}.${NAMESPACE}.svc.cluster.local

# Individual pod DNS for StatefulSets
# Format: ${POD_NAME}.${HEADLESS_SVC}.${NAMESPACE}.svc.cluster.local
kubectl exec -n ${NS} ${POD} -- nslookup ${STATEFULSET_POD}-0.${HEADLESS_SVC}.${NAMESPACE}.svc.cluster.local

Interpreting Results

| Result | Meaning | |--------|---------| | Returns IP address | Resolution working | | ** server can't find ... NXDOMAIN | Name does not exist — wrong service name, namespace, or missing service | | bad address / could not resolve host | DNS lookup failed — CoreDNS unreachable or misconfigured | | connection timed out; no servers could be reached | Cannot reach CoreDNS — network or service issue | | SERVFAIL | CoreDNS received the query but failed to resolve — check Corefile and upstream |

---

Section 3: CoreDNS Configuration (Corefile)

The Corefile is stored as a ConfigMap and defines how CoreDNS handles queries.

# View the Corefile
kubectl get configmap coredns -n kube-system -o jsonpath='{.data.Corefile}'

# View with line numbers for reference
kubectl get configmap coredns -n kube-system -o jsonpath='{.data.Corefile}' | cat -n

# Check if a custom ConfigMap exists
kubectl get configmap coredns-custom -n kube-system 2>/dev/null

Default Corefile Structure

A healthy default Corefile looks like:

.:53 {
    errors
    health {
       lameduck 5s
    }
    ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {
       pods insecure
       fallthrough in-addr.arpa ip6.arpa
       ttl 30
    }
    prometheus :9153
    forward . /etc/resolv.conf
    cache 30
    loop
    reload
    loadbalance
}

Key Plugins and What Can Go Wrong

| Plugin | Purpose | Common Issue | |--------|---------|-------------| | kubernetes | Resolves cluster.local names | Wrong cluster domain, pods setting, or missing fallthrough | | forward | Sends non-cluster queries upstream | Wrong upstream address, unreachable forwarder | | cache | Caches responses | TTL too high causes stale answers; too low overloads upstream | | loop | Detects forwarding loops | CoreDNS restarts if loop detected — check /etc/resolv.conf on nodes | | errors | Logs errors | Missing plugin hides failures | | health | Liveness endpoint on :8080 | If missing, kubelet cannot health-check CoreDNS | | ready | Readiness endpoint on :8181 | If missing, endpoints not populated until pod is actually ready | | reload | Hot-reload on Corefile change | Without this, ConfigMap changes require pod restart |

Common Corefile Misconfigurations

| Problem | Symptom | Diagnostic | |---------|---------|-----------| | Wrong cluster domain | All internal lookups return NXDOMAIN | Check if kubernetes zone matches --cluster-domain kubelet flag | | forward . 8.8.8.8 when nodes use private DNS | External resolution of private zones fails | Check if forward directive should use /etc/resolv.conf or a private upstream | | Missing loop plugin | CoreDNS forwards to itself via node resolv.conf, infinite loop | Check if loop plugin is present in Corefile | | pods verified instead of pods insecure | Pod A/AAAA records fail verification checks | Check if pods setting is verified — recommend pods insecure unless strict verification is needed | | Custom forward zone typo | Queries for that zone fail | Verify zone block spelling and upstream IP | | Missing fallthrough in kubernetes block | Reverse lookups for pod IPs fail | Che

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: foxj77
• Source: foxj77/claude-code-skills
• License: MIT

Install and usage instructions live in the source repository linked above.