External Dns Troubleshooting

ExternalDNS Troubleshooting

Diagnose failures in ExternalDNS — the controller that synchronises Kubernetes Service/Ingress annotations to DNS providers (Route53, Azure DNS, CloudFlare, Google Cloud DNS, etc.).

Keywords

external-dns, externaldns, dns, records, route53, azure-dns, cloudflare, google-cloud-dns, dns-sync, dns-records, a-record, cname, txt-record, annotation, hostname, domain, zone, provider, registry, ownership

When to Use This Skill

• DNS records are not being created for new Services or Ingresses
• Existing DNS records are stale, pointing to old IPs
• ExternalDNS logs show authentication, permission, or zone errors
• TXT ownership records are missing or conflicting
• Records appear in the log as "planned" but never created
• DNS propagation seems broken after a provider migration
• ExternalDNS is running but doing nothing (no changes detected)

When NOT to Use

• DNS records exist but TLS certs are failing → use [cert-manager-troubleshooting](../cert-manager-troubleshooting)
• DNS is working but the Service has no endpoints → use [k8s-namespace-troubleshooting](../k8s-namespace-troubleshooting)
• ExternalDNS config is delivered by Flux and not appearing → use [flux-troubleshooting](../flux-troubleshooting)

Related Skills

• [cert-manager-troubleshooting](../cert-manager-troubleshooting) - TLS certificates depend on DNS
• [kyverno-troubleshooting](../kyverno-troubleshooting) - Policies may block annotations
• [external-secrets-troubleshooting](../external-secrets-troubleshooting) - Provider credentials from external stores
• [k8s-namespace-troubleshooting](../k8s-namespace-troubleshooting) - General namespace diagnosis
• [k8s-platform-operations](../k8s-platform-operations) - Cluster-wide operations
• [flux-troubleshooting](../flux-troubleshooting) - GitOps delivery of ExternalDNS config

Quick Reference

| Task | Command | |------|---------| | Check ExternalDNS pod | kubectl get pods -n external-dns | | View logs | kubectl logs -n external-dns deploy/external-dns --tail=200 | | List managed records (dry-run) | kubectl logs -n external-dns deploy/external-dns \| grep "Desired" | | Check source annotations | kubectl get ingress -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}\t{.metadata.annotations.external-dns\.alpha\.kubernetes\.io/hostname}\n{end}' | | Find TXT ownership records | kubectl logs -n external-dns deploy/external-dns \| grep "txt" | | Force sync (recommend to user) | kubectl rollout restart deploy/external-dns -n external-dns |

---

Diagnostic Workflow

DNS record not appearing?
├─ ExternalDNS pod running?
│   ├─ No → Check deployment, image, RBAC (Section 1)
│   └─ Yes → Check logs for errors
│       ├─ Authentication/permission error → Fix provider credentials (Section 2)
│       ├─ Zone not found / filtered out → Fix domain filter config (Section 3)
│       ├─ Source yielded 0 endpoints → Fix annotations/sources (Section 4)
│       ├─ Record planned but not created → Provider API issue (Section 5)
│       └─ No log activity at all → Check interval and RBAC (Section 6)
├─ Record exists but wrong value?
│   ├─ TXT ownership record missing → Ownership conflict (Section 7)
│   └─ Multiple ExternalDNS instances → Registry conflict (Section 7)
└─ Record was deleted unexpectedly?
    └─ Orphaned record cleanup or ownership loss (Section 7)

---

Section 1: Controller Health

# Pod status
kubectl get pods -n external-dns -o wide
kubectl describe deploy/external-dns -n external-dns

# RBAC — ExternalDNS needs read access to Services, Ingresses, Nodes
kubectl auth can-i list services --as=system:serviceaccount:external-dns:external-dns
kubectl auth can-i list ingresses --as=system:serviceaccount:external-dns:external-dns

# Check arguments and configuration
kubectl get deploy/external-dns -n external-dns -o jsonpath='{.spec.template.spec.containers[0].args}' | tr ',' '\n'

Key Arguments to Verify

| Argument | Purpose | Common Mistake | |----------|---------|----------------| | --source=service,ingress | What resources to watch | Missing a source type | | --domain-filter=example.com | Restrict to specific domains | Typo in domain or too restrictive | | --provider=aws | DNS provider | Wrong provider name | | --policy=upsert-only | Create/update but never delete | Using sync deletes unmanaged records | | --registry=txt | Ownership tracking | Missing registry causes conflicts | | --txt-owner-id=my-cluster | Unique owner per cluster | Duplicate owner IDs across clusters | | --interval=1m | Sync interval | Too long hides issues |

---

Section 2: Provider Authentication

> Full provider auth reference: See [provider-authentication.md](../_shared/references/provider-authentication.md) for generic diagnostic steps, auth method tables, and provider-specific issue matrices for AWS, Azure, GCP, Vault, and CloudFlare.

The commands below are ExternalDNS-specific. For general auth debugging patterns, use the shared reference.

# Quick auth check — grep ExternalDNS logs for auth failures
kubectl logs -n external-dns deploy/external-dns --tail=200 | grep -iE 'credential|auth|forbidden|access denied|assume role|sts|azure|authorization|cloudflare|google|permission'

Required Provider Permissions (ExternalDNS-Specific)

| Provider | Required Permissions | |----------|---------------------| | AWS Route53 | route53:ChangeResourceRecordSets, route53:ListResourceRecordSets, route53:ListHostedZones, route53:ListHostedZonesByName | | Azure DNS | DNS Zone Contributor on the zone resource group (or custom role with Microsoft.Network/dnsZones/*) | | CloudFlare | API token with Zone:DNS:Edit on target zones | | Google Cloud DNS | roles/dns.admin on the project (or dns.changes.create, dns.resourceRecordSets.*, dns.managedZones.list) |

---

Section 3: Domain Filter and Zone Issues

# Check which domains ExternalDNS is configured to manage
kubectl get deploy/external-dns -n external-dns -o jsonpath='{.spec.template.spec.containers[0].args}' | tr ',' '\n' | grep -i domain

# Check if the target zone exists at the provider
kubectl logs -n external-dns deploy/external-dns --tail=200 | grep -iE 'zone|domain|filter'

# List all zones ExternalDNS can see
kubectl logs -n external-dns deploy/external-dns --tail=200 | grep -i "all hosted zones"

| Problem | Symptom in Logs | Diagnostic | |---------|-----------------|-----------| | Domain filter too strict | "0 zones match" or skipped records | Check --domain-filter scope; recommend widening or adding --exclude-domains | | Zone doesn't exist at provider | "zone not found" | Verify zone exists at the provider | | Zone ID filter wrong | Records in wrong zone | Check --zone-id-filter argument against provider zone IDs | | Sub-domain not covered | Parent zone exists but child not matched | Check if sub-domain is included in domain filter |

---

Section 4: Source and Annotation Issues

# Check which sources ExternalDNS watches
kubectl get deploy/external-dns -n external-dns -o jsonpath='{.spec.template.spec.containers[0].args}' | tr ',' '\n' | grep source

# Find Services with external-dns annotations
kubectl get svc -A -o json | jq -r '.items[] | select(.metadata.annotations["external-dns.alpha.kubernetes.io/hostname"] != null) | "\(.metadata.namespace)/\(.metadata.name)\t\(.metadata.annotations["external-dns.alpha.kubernetes.io/hostname"])"'

# Find Ingresses with hostnames
kubectl get ingress -A -o json | jq -r '.items[] | select(.spec.rules) | .metadata.namespace + "/" + .metadata.name + "\t" + (.spec.rules[].host // "no-host")'

# Check if source yields endpoints
kubectl logs -n external-dns deploy/external-dns --tail=200 | grep -i "endpoint"

Required Annotations (Services)

| Annotation | Purpose | Example | |------------|---------|---------| | external-dns.alpha.kubernetes.io/hostname | Target DNS name | app.example.com | | external-dns.alpha.kubernetes.io/ttl | Record TTL | "300" | | external-dns.alpha.kubernetes.io/target | Override target IP/hostname | lb.example.com |

Service Type Requirements

| Service Type | ExternalDNS Behaviour | |--------------|----------------------| | LoadBalancer | Uses .status.loadBalancer.ingress for target IP/hostname | | ClusterIP | Only works if --publish-internal-services is set | | NodePort | Only works if --service-type-filter=NodePort and node IPs available | | ExternalName | Uses .spec.externalName as CNAME target |

Common issue: LoadBalancer Service has no .status.loadBalancer.ingress yet — ExternalDNS cannot determine the target IP. Check if the cloud load balancer provisioned successfully.

---

Section 5: Record Creation Failures

# Check for planned vs applied changes
kubectl logs -n external-dns deploy/external-dns --tail=300 | grep -iE 'planned|create|update|delete|change|apply'

# Check for rate limiting
kubectl logs -n external-dns deploy/external-dns --tail=300 | grep -iE 'throttl|rate limit|429|too many'

# Check for provider API errors
kubectl logs -n external-dns deploy/external-dns --tail=300 | grep -iE 'error|fail|invalid|conflict'

| Problem | Symptom | Diagnostic | |---------|---------|-----------| | Planned but not applied | "Creating" in logs, record missing at provider | Check if --dry-run flag is set in deployment args | | Rate limited | 429 errors or "rate limit" messages | Check --interval value and record count | | Invalid record | "invalid" or "validation" errors | Check record name/value format for the provider | | Conflicting record | "conflict" or "already exists" | TXT ownership mismatch — see Section 7 |

---

Section 6: No Activity

# Check sync interval
kubectl get deploy/external-dns -n external-dns -o jsonpath='{.spec.template.spec.containers[0].args}' | tr ',' '\n' | grep interval

# Check if watching the right namespace
kubectl get deploy/external-dns -n external-dns -o jsonpath='{.spec.template.spec.containers[0].args}' | tr ',' '\n' | grep namespace

# Check RBAC can list resources
kubectl auth can-i list services -A --as=system:serviceaccount:external-dns:external-dns
kubectl auth can-i list ingresses -A --as=system:serviceaccount:external-dns:external-dns

# Recent log entries (any activity at all?)
kubectl logs -n external-dns deploy/external-dns --tail=50 --since=10m

| Cause | Check | Diagnostic | |-------|-------|-----------| | --namespace flag restricts scope | Args show --namespace=X | Check if namespace filter is intentional or too restrictive | | RBAC too restrictive | auth can-i returns "no" | Check ClusterRole permissions for services/ingresses/nodes | | No annotated resources exist | Services/Ingresses lack annotations | Check if external-dns.alpha.kubernetes.io/hostname annotation is present | | Interval too long | Args show --interval=60m | Check --interval value; recommend reducing for faster feedback |

---

Section 7: Ownership and Registry

ExternalDNS uses TXT records as an ownership registry to track which records it manages. Problems here cause records to be ignored, duplicated, or accidentally deleted.

# Check registry type and owner ID
kubectl get deploy/external-dns -n external-dns -o jsonpath='{.spec.template.spec.containers[0].args}' | tr ',' '\n' | grep -iE 'registry|owner'

# Look for ownership conflicts in logs
kubectl logs -n external-dns deploy/external-dns --tail=300 | grep -iE 'ownership|registry|txt|skip|ignore|conflict'

| Problem | Cause | Diagnostic | |---------|-------|-----------| | Record exists but ExternalDNS ignores it | TXT record has different owner ID | Compare --txt-owner-id with TXT record owner; check for mismatch | | Multiple clusters fighting over records | Same --txt-owner-id on different clusters | Check if each cluster has a unique owner ID | | Records deleted unexpectedly | --policy=sync removes unmanaged records | Check if --policy=sync is set; recommend upsert-only if unintended | | Orphaned records after cluster deletion | No cleanup ran | Check provider console for orphaned DNS and TXT ownership records | | TXT prefix collision | Default prefix a- or cname- collides | Check --txt-prefix for uniqueness |

---

Provider-Specific Gotchas

| Provider | Gotcha | Detail | |----------|--------|--------| | Route53 | Private hosted zones | Need --aws-zone-type=private and VPC association | | Route53 | Alias records | ExternalDNS creates A records, not aliases — use --aws-prefer-cname for CNAME | | Azure DNS | Resource group required | Must specify --azure-resource-group | | Azure DNS | Subscription filter | Multi-subscription needs --azure-subscription-id | | CloudFlare | Proxy mode | external-dns.alpha.kubernetes.io/cloudflare-proxied: "true" for orange cloud | | CloudFlare | Zone ID vs name | API token must have access to the specific zone | | Google Cloud DNS | Project required | Must specify --google-project | | Google Cloud DNS | Managed zone name | Zone names differ from domain names |

---

MCP Tools Available

When the appropriate MCP servers are connected, prefer these over raw kubectl where available:

• mcp__flux-operator-mcp__get_kubernetes_resources - Query ExternalDNS deployment, pods, services, ingresses
• mcp__flux-operator-mcp__get_kubernetes_logs - Retrieve ExternalDNS pod logs
• mcp__flux-operator-mcp__get_kubernetes_metrics - Check ExternalDNS resource consumption

---

Common Mistakes

| Mistake | Why It Fails | Instead | |---------|--------------|---------| | Running --policy=sync without understanding it | Deletes DNS records not managed by ExternalDNS, causing outages | Start with upsert-only; switch to sync only after auditing all existing records | | Same --txt-owner-id across multiple clusters | Clusters overwrite each other's records | Use a unique owner ID per cluster (e.g., cluster-name-region) | | Forgetting --source=ingress when using Ingresses | ExternalDNS only watches Services by default in some configs | Explicitly list all source types: --source=service --source=ingress | | Setting --dry-run and forgetting to remove it | Records are planned but never created — looks like a provider issue | Check args for --dry-run before investigating provider auth | | Not checking LoadBalancer provisioning | ExternalDNS has no target IP to use | Verify .status.loadBalancer.ingress is populated on the Service | | Debugging DNS propagation before checking ExternalDNS logs | Wastes time on DNS caching when the record was never created | Always check ExternalDNS logs first, then provider console, then DNS propagation |

Behavioural Guidelines

1. Check logs first — ExternalDNS is heavily log-driven; the answer is almost always in the logs.
2. Verify annotations — Most "ExternalDNS isn't working" issues are missing or malformed annotations.
3. Check the provider console — Confirm whether the record exists at the provider, not just via DNS lookup (caching).
4. Never expose provider credentials — List secret names, never decode values.
5. Understand the policy — upsert-only vs sync has dramatically different behaviour. Confirm which is set before troubleshooting deletions.
6. Check owner IDs in multi-cluster setups — Ownership conflicts are silent; records just stop updating.
7. Restart as a last resort — A rollout restart forces a full sync cycle, but diagnose the root cause first.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: foxj77
• Source: foxj77/claude-code-skills
• License: MIT

Install and usage instructions live in the source repository linked above.