CoreDNS Monitoring and Performance Optimization Complete Guide
In Amazon EKS and modern Kubernetes clusters, CoreDNS is the core component responsible for all in-cluster service discovery and external domain name resolution. Since CoreDNS performance and availability directly impact application response times and stability, building an effective monitoring and optimization architecture is critical. This article analyzes CoreDNS performance monitoring metrics, TTL configuration guide, monitoring architecture best practices, and AWS operational checks and diagnostic examples. Each section leverages Prometheus metrics and Amazon EKS environment examples.
1. CoreDNS Performance Monitoring: Key Prometheus Metrics
CoreDNS exposes Prometheus-format metrics with prometheus :9153 in the Corefile. Port 9153 is the scrape example in this guide; verify the Pod listener and collection configuration. Do not assume the kube-dns Service exposes it. Core metrics cover DNS throughput, latency, response codes, and caching efficiency. Plugin and EKS references are listed under Primary sources.
CoreDNS 4 Golden Signals
CoreDNS Key Prometheus Metrics
Examples assume one cluster scraped as job="coredns". Use your actual job name and add a cluster selector when querying multiple clusters. Verify metric and label names on the scraped /metrics endpoint before use.
| Metric | Signal | Description / PromQL |
|---|---|---|
coredns_dns_requests_totalCounter | Traffic | DNS QPS across the selected scrape scope, summed over servers, Pods, and protocols. |
coredns_dns_request_duration_secondsHistogram | Latency | P99 of the combined request distribution. Sum buckets by le rather than averaging Pod quantiles. |
coredns_dns_responses_totalCounter | Response codes | Response QPS by rcode. Track SERVFAIL and distinguish expected negative answers from unexpected NXDOMAIN responses. |
coredns_cache_hits_totalCounter | Cache | Aggregate cache hit ratio (0–1). The ratio is undefined without requests; evaluate it with a traffic condition. |
coredns_cache_requests_totalCounter | Cache | Cache miss QPS = requests minus hits, replacing the deprecated cache_misses_total counter. |
coredns_proxy_request_duration_seconds_count{proxy_name="forward"}Histogram count | Forward | Forward proxy request QPS by upstream, preserving the to label. |
coredns_proxy_request_duration_seconds_count{proxy_name="forward",rcode="SERVFAIL"}Histogram count | Forward errors | SERVFAIL QPS by upstream. This is not an all-error count; inspect other rcodes separately. |
coredns_panics_totalCounter | Stability | New panics over the last five minutes, distinguishing current incidents from historical totals. |
Metric names follow the current forward and cache references. Replace coredns_forward_requests_total and coredns_forward_responses_total with the proxy histogram's _count, and derive cache misses from requests minus hits. Check which metrics your running CoreDNS version exposes.
Beyond these, additional metrics such as request/response size (coredns_dns_request_size_bytes, ...response_size_bytes), DO bit presence (coredns_dns_do_requests_total), and plugin-specific metrics are available. For example, the Forward plugin provides upstream query time (coredns_proxy_request_duration_seconds{proxy_name="forward"}), and the kubernetes plugin provides API update latency (coredns_kubernetes_dns_programming_duration_seconds).
Key Metric Meanings and Usage
Track coredns_dns_requests_total per-second rate for DNS QPS, distributed per CoreDNS Pod to verify load balance. If QPS consistently grows, evaluate scale-out. When p99 latency rises above normal, check upstream delays and CPU/memory saturation. A low cache hit ratio requires checking query reuse, TTLs, and evictions together. SERVFAIL indicates a resolution failure; REFUSED can reflect policy or the forward max_concurrent limit. NXDOMAIN can be expected for nonexistent names and search-domain expansion. Compare response-code baselines, application failures, and narrowly scoped query logs before treating an increase as an incident or an incorrect lookup.
Monitor CPU and memory against the installed EKS add-on version, configuration, and Deployment resources. Do not assume 70Mi/170Mi are universal EKS memory defaults. If the actual limit is 170Mi, an alert at 150Mi is one trial threshold for that deployment. Account for growth, OOM history, cache capacity, and Kubernetes object counts. Compare CPU usage with the configured limit and observed throttling.
Traffic to link-local services such as Amazon DNS shares a 1024 PPS per ENI allowance with IMDS and Amazon Time Sync. This is not a limit on all DNS traffic. Increasing max_concurrent does not increase it. Monitor linklocal_allowance_exceeded from the AWS ENA metrics.
2. CoreDNS TTL Configuration Guide and Amazon EKS Examples
TTL (Time-To-Live) defines the DNS record cache validity period, balancing DNS traffic load and information freshness. CoreDNS handles TTL at two levels:
The plugin defaults below are scoped to upstream CoreDNS v1.11.3, using the kubernetes reference and cache reference. They do not imply that an EKS add-on uses the same configuration.
- Kubernetes response TTL: The TTL of records generated by the
kubernetesplugin, not just SOA records. Omittingttlgives 5 seconds; the allowed range is 0–3600 seconds. When using TTL 0 to avoid caching, also check that caches along the path respect zero. - Cache TTL: The
cacheplugin defaults to a maximum of 3600 seconds for success and 1800 seconds for denial, with a 5-second minimum TTL (MINTTL) for each. Denial includes NXDOMAIN and NODATA responses.cache 30sets both ceilings to 30 seconds;success CAPACITY TTL MINTTLanddenial CAPACITY TTL MINTTLoverride them separately. SERVFAIL caching has a separateservfailoption, defaulting to 5 seconds in this version.
For ordinary success/denial entries, the effective cache lifetime is min(max(TTL derived from the response, MINTTL), maximum TTL) (v1.11.3 implementation). Thus a 5-second record + cache 30 + the default 5-second minimum = 5 seconds, not 30 seconds. Explicitly raising the minimum with success 9984 30 10 permits caching that successful response for 10 seconds. Capacity pressure can evict entries earlier; enabling serve_stale separately changes the policy for serving expired answers.
Amazon EKS Default CoreDNS Configuration
Check the installed add-on version, Corefile, and Deployment together. With no ttl in the kubernetes block and only cache 30, internal records remain valid in cache for 5 seconds under the version described above. With an explicit ttl 30, the internal record TTL itself is 30 seconds. Long external TTLs are capped; short TTLs are evaluated against the minimum as well.
The kubernetes plugin watches Kubernetes API objects and maintains local state. A DNS cache miss does not imply another API request, so increasing DNS cache TTL should not be described as preventing per-query Kubernetes API lookups.
TTL Configuration Guide
Short TTLs can propagate changes sooner but increase repeat queries; longer TTLs can reduce repeat queries while extending use of old addresses. 30 seconds is a trial value in this guide, not a universal optimum. Choose values using service/endpoint churn, failover objectives, and measured cache hit ratios. Evaluate denial TTL separately because it can delay discovery of newly created names or records.
For prefetch AMOUNT [[DURATION] [PERCENTAGE%]], prefetch 5 60s 10% identifies popular entries using a threshold of five queries without gaps of 60 seconds or more. On a cache hit, refresh is considered when the remaining TTL reaches the 10% threshold. 60 seconds is not the time before expiry to refresh. In v1.11.3, omitted DURATION defaults to one minute and omitted PERCENTAGE to 10%, with a range of 10–90%. The reference's one-second-before-expiry boundary is also evaluated on subsequent queries; this is not a timer that runs without traffic.
Amazon EKS Application Examples
If a workload tolerates using internal answers for up to 30 seconds, trial ttl 30 in the kubernetes block together with the cache settings. Client, NodeLocal DNSCache, and application-runtime caching policies also affect the result. For the EKS managed add-on, check the installed version's supported schema and manage custom Corefiles through configurationValues.corefile. Direct ConfigMap edits can be overwritten during add-on updates (EKS management guide).
For Aurora or similar endpoints, check the actual DNS response TTL and connection-pool behavior. If an observed endpoint returns a 1-second TTL and the cache ceiling is 30 seconds, the v1.11.3 default 5-second minimum can extend it to 5 seconds. Do not assume every Aurora endpoint has a 1-second TTL.
In a server block for a verified DNS zone that needs low TTLs, trial an explicit zero minimum such as success 9984 30 0 and denial 2048 10 0. The first number is capacity, not TTL: success 1 and denial 1 do not mean one-second caching. Check the entire path, including NodeLocal DNSCache, and avoid broadening an exception to all of amazonaws.com.
3. CoreDNS Monitoring Architecture Best Practices
Metric Collection and Storage
Two common approaches in Amazon EKS:
- Amazon Managed Service for Prometheus (AMP): Fully managed Prometheus-compatible service. Install ADOT Collector or Prometheus to scrape and forward CoreDNS metrics.
- CloudWatch Container Insights: Configure the CloudWatch agent's Prometheus collection to scrape CoreDNS Pod metrics endpoints. Verify application-metric targets and ports separately.
First verify the Corefile prometheus listener, Service ports, and labels. If a Service exposes the metrics port, select that named port in the ServiceMonitor; otherwise configure a dedicated metrics Service or Pod-based discovery. The k8s-app=kube-dns label alone does not guarantee that port 9153 is exposed.
Log Collection
Enable log or errors plugins as needed. Use Fluent Bit or Fluentd DaemonSets to collect CoreDNS stdout/stderr logs and export to CloudWatch Logs.
Avoid excessive logging overhead. Set metadata caching (Kube_Meta_Cache_TTL=60) and reduce unnecessary field collection.
Visualization and Dashboards
Use Grafana (or Amazon Managed Grafana) to visualize CoreDNS metrics: QPS, latency histograms, error rates (rcode distribution), cache hit rates.
Alerting
Use Prometheus alerting rules with Alertmanager, or CloudWatch Alarms:
- CoreDNSDown: Detect all selected
up{job="coredns"}targets being down or absent. This measures scrape availability; use a separate DNS query probe to check resolution availability. - HighDNSLatency: The appendix trials p99 above 50ms for five minutes while traffic is present. Adjust it to the workload baseline and SLO.
- DNSErrorsSpike: Track the
SERVFAILresponse ratio. InvestigateREFUSEDagainst intended policy and concurrency limits, andNXDOMAINagainst its baseline and application impact. - ENIThrottling: Increases in
linklocal_allowance_exceededindicate dropped link-local service packets, including Amazon DNS traffic. - HighCoreDNSCPU/Memory: Resource utilization alerts
4. Amazon EKS Best Practices and Customer Cases
The following are operational checks and diagnostic scenarios based on public CoreDNS and AWS documentation. They are not measured customer case studies; validate causes in the installed configuration and workload.
CoreDNS Horizontal Scaling
Verify replicas in the current Deployment and add-on configuration. Evaluate supported EKS managed add-on autoscaling or Cluster Proportional Autoscaler using node count, CPU cores, and measured query load. Avoid two controllers managing the same replica count.
NodeLocal DNSCache
For large clusters or high DNS traffic, deploy NodeLocal DNSCache DaemonSet for local DNS caching on supported nodes.
DNS Packet Limits and Traffic Distribution
Link-local traffic to Amazon DNS shares a 1024 PPS/ENI allowance. Co-located CoreDNS Pods can concentrate upstream traffic on one ENI. Spread Pods and reduce repeat queries through caching; NodeLocal DNSCache still sends upstream queries on misses and does not remove ENI limits.
Graceful Termination (Lameduck & Ready Plugin)
health { lameduck DURATION } delays shutdown. During that interval, /health still returns 200 while /ready no longer reports readiness (health reference). Upstream v1.11.3 adds no delay when lameduck is omitted. 30 seconds is not a universal AWS recommendation, and the appendix's 5 seconds is also a trial value. Measure endpoint propagation and shutdown time; budget terminationGracePeriodSeconds for preStop, lameduck, and shutdown with headroom.
The EKS guide identifies /ready probes in v1.9.3-eksbuild.3 and later and v1.10.1-eksbuild.6 and later. Custom Corefiles for those configurations must include ready. Check the actual add-on version and probe configuration together.
Higher QPS Requirements
- Evaluate
max_concurrent: upstream v1.11.3 imposes no concurrency cap through this option when it is omitted. Size an explicit cap from upstream QPS × latency plus headroom. The appendix's2000is illustrative; account for roughly 2 KB per concurrent query andREFUSEDresponses when the cap is exceeded (forward reference). - Scale CoreDNS horizontally or deploy NodeLocal DNSCache
- Collect increases in ENA
linklocal_allowance_exceeded. Publishing it to CloudWatch requires agent configuration; verify the emitted metric name and dimensions.
Key Summary
These are examples to validate against your workload and SLO, not measured benchmarks or universal EKS recommendations.
| Metric | Example target | Example investigation trigger | Scope |
|---|---|---|---|
| Query Latency (P99) | < 50ms | > 50ms / 5m | Trial threshold used in the appendix. Evaluate with traffic and adjust to the SLO |
| Throughput (QPS/Pod) | Determine by load test | Growth with latency/saturation | Low QPS alone is not a failure. Capacity depends on query mix and resources |
| Cache Hit Ratio | > 80% | < 50% | Trial values for workloads with repeated queries. TTL 30s does not guarantee this hit ratio |
| Error Rate (SERVFAIL) | < 0.1% | > 1% | Trial SERVFAIL ratio over all responses. Excludes expected NXDOMAIN |
| CPU Utilization | < 60% | > 80% | Percent of the configured CPU limit, if present. Evaluate alongside actual throttling |
| Memory Utilization | < 120Mi | > 150Mi | Example only for a deployment with an actual 170Mi limit. Not an EKS default or measured result |
Plugin baseline sources: forward v1.11.3 · health v1.11.3
| Parameter | Baseline scope | Trial value | Scope |
|---|---|---|---|
max_concurrent | v1.11.3: no cap when omitted | 2000 | Trial value. Size from upstream QPS × latency plus headroom; allow roughly 2KB per query |
Replica Count | Inspect current Deployment | Auto-proportional | Evaluate supported EKS add-on autoscaling or Cluster Proportional Autoscaler |
lameduck | v1.11.3: no delay when omitted | 5s | Trial value. Check /ready and endpoint propagation; allow headroom in the Pod grace period |
Benchmark tool: dnsperf -s <COREDNS_IP> -d queries.txt -c 10 -T 10 to measure CoreDNS QPS and latency.
- Monitoring Metrics: Requests, latency, cache requests/hits, rcode distribution, and CPU/memory. Derive cache misses from requests minus hits.
- Trial TTLs: Internal answers 30s, success ceiling 30s, denial ceiling 10s,
prefetch 5 60s 10%. Choose values from freshness objectives and measurements;cache 30alone does not cache a 5-second record for 30 seconds. - Monitoring: Grafana dashboards matched to the scrape configuration + Prometheus alerting rules, NodeLocal DNSCache for scale-out
Appendix: Configuration Examples
Recommended Corefile
This is a Corefile example requiring workload validation, not a universal replacement for an EKS Corefile. Verify the 30-second internal-answer freshness budget, cache capacity, upstream concurrency, and shutdown grace period. Capacities are rounded down to multiples of 256, so this example uses 9984 and 2048. Enable query log only for a scoped diagnostic period.
.:53 {
errors
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
ttl 30 # Trial response TTL; requires a 30s freshness budget
}
cache 30 { # Ceiling, not a forced lifetime for every response
success 9984 30 5 # Capacity, maximum TTL, minimum TTL (seconds)
denial 2048 10 5 # Separate negative-cache capacity and TTL bounds
prefetch 5 60s 10% # Popularity: gaps <60s; refresh at remaining-TTL threshold
}
forward . /etc/resolv.conf {
max_concurrent 2000 # Trial cap; size from upstream QPS and latency
}
prometheus :9153
health {
lameduck 5s # Trial delay; allow shutdown headroom in the Pod grace period
}
ready
loop
reload
loadbalance
}
Alertmanager Rule Examples
This YAML is a Prometheus rule file whose alerts are sent to Alertmanager. Like the metric table, it assumes one cluster scraped as job="coredns". Substitute the real job name and add the same cluster condition to every selector for a multi-cluster backend. Verify /metrics and scrape labels first. Ratios, latency, and durations are trial thresholds. Expected NXDOMAIN is excluded from the critical error alert; evaluate REFUSED separately against policy and concurrency limits. Response-code ratios cannot detect unanswered queries or unavailable Pods alone, so check scrape state and actual DNS query probes as well.
groups:
- name: coredns-example
rules:
- alert: CoreDNSHighErrorRate
expr: >
(sum(rate(coredns_dns_responses_total{job="coredns",rcode="SERVFAIL"}[5m])) /
sum(rate(coredns_dns_responses_total{job="coredns"}[5m]))) > 0.01
and on() sum(rate(coredns_dns_responses_total{job="coredns"}[5m])) > 0
for: 10m
labels:
severity: critical
annotations:
description: "CoreDNS SERVFAIL response ratio > 1% for 10 min"
- alert: CoreDNSP99Latency
expr: >
histogram_quantile(0.99,
sum by (le) (rate(coredns_dns_request_duration_seconds_bucket{job="coredns"}[5m]))) > 0.05
and on() sum(rate(coredns_dns_requests_total{job="coredns"}[5m])) > 0
for: 5m
labels:
severity: warning
- alert: CoreDNSScrapeUnavailable
expr: sum(up{job="coredns"}) == 0 or absent(up{job="coredns"})
for: 5m
labels:
severity: warning
annotations:
description: "No healthy CoreDNS scrape target; check collection and DNS probes"
Large Clusters (>100 Nodes or QPS > 5k)
- NodeLocal DNSCache (DaemonSet) for local caching and RTT reduction
- CloudWatch Container Insights as alternative when Prometheus collection is difficult
Primary sources
- CoreDNS v1.11.3 cache reference, TTL implementation, prefetch implementation
- CoreDNS v1.11.3 kubernetes reference
- CoreDNS v1.11.3 health reference, health implementation, forward reference, forward implementation
- CoreDNS Prometheus plugin
- Amazon EKS: Manage CoreDNS
- Amazon EC2: ENA network performance metrics, link-local PPS limit
- Kubernetes NodeLocal DNSCache
- Prometheus alerting rules, query functions