Kubernetes DNS and Service Discovery Troubleshooting

Kubernetes DNS is the in-cluster name-resolution layer that lets Pods discover Services by stable names instead of tracking changing Pod IP addresses.

For small teams, most Kubernetes DNS incidents become easier once the failure is separated into the right layer. A short Service name may be wrong for the caller's namespace, the Service may exist without usable endpoints, a Pod may have unexpected resolver settings, CoreDNS may be unhealthy, or a NetworkPolicy may block the DNS path. Raff Technologies treats these as separate checks rather than one generic “networking problem.”

This guide focuses on that diagnostic boundary. It explains how Service names are formed, how namespaces affect lookup, what CoreDNS actually provides, how Service and EndpointSlice failures can look like DNS failures, and how to distinguish cluster DNS problems from application-connectivity problems.

Kubernetes DNS turns Service identity into stable names

Pods are disposable. Their IP addresses can change as workloads roll out, restart, scale, or move between nodes. Kubernetes Services provide a stable identity in front of those Pods, while cluster DNS makes that identity available by name.

A simplified request path looks like this:

Application Pod
      ↓ DNS query
Cluster DNS
      ↓ resolves
Service name
      ↓ stable Service identity
Eligible endpoints
      ↓
Application Pods

For a normal Service, DNS records resolve to the Service's cluster IP. For a headless Service, DNS can return the addresses of the backing Pods instead. That difference is expected behavior, not evidence that DNS is inconsistent.

Kubernetes also creates SRV records for named Service ports, allowing clients that understand SRV lookups to discover both a port and the relevant Service target.

The important boundary is this: DNS answers the naming question; it does not prove the application behind that name is healthy.

A Service name can resolve successfully while the connection still fails because:

• the Service selector matches no ready Pods;
• EndpointSlices contain no usable endpoints;
• the target port is wrong;
• NetworkPolicy blocks the traffic;
• the application is not listening;
• readiness removes Pods from eligible endpoints;
• the network path fails after name resolution.

This is why a successful nslookup is only one part of a connectivity diagnosis.

Service discovery depends on namespaces and DNS search paths

Kubernetes Service names are namespace-aware.

A Pod can normally resolve a Service in its own namespace by using the short Service name:

api

A Pod in another namespace should normally qualify the name with the target namespace:

api.production

The fully qualified form is commonly:

api.production.svc.cluster.local

cluster.local is a common default cluster domain, but clusters can use another domain. Do not hard-code it into application logic without knowing the cluster's actual DNS configuration.

Kubernetes configures a Pod's resolver search path so short names can be expanded automatically. A typical /etc/resolv.conf inside a Pod includes search domains similar to:

<namespace>.svc.cluster.local
svc.cluster.local
cluster.local

The exact values depend on cluster configuration.

This creates one of the most common debugging patterns: a short name fails while a namespace-qualified or fully qualified name works. That usually points to a naming or resolver-search issue rather than a broken Service.

A compact check is:

cat /etc/resolv.conf

Then compare what the application is requesting with the namespace where the Service actually exists.

At Raff, the DNS troubleshooting rule is to separate naming from reachability: verify the expected name and namespace first, then the Service and EndpointSlices, then cluster DNS health, and only after that the underlying network path.

That order avoids changing firewalls or cluster networking to solve what is actually a namespace typo.

DNS troubleshooting starts by separating resolution from connectivity

The first useful question is not “Is Kubernetes networking broken?” It is what exactly failed?

A diagnostic matrix makes the failure layer visible.

A simple lookup against the built-in Kubernetes Service is often useful because it tests cluster DNS independently from your application Service:

nslookup kubernetes.default

If that works but api.production does not, the investigation can move toward the specific Service and namespace instead of immediately treating CoreDNS as unhealthy.

If both fail from one Pod, inspect that Pod's resolver settings. If they fail from many unrelated Pods, cluster DNS becomes a stronger suspect.

Raff's troubleshooting framework separates four layers: name, Service endpoints, cluster DNS, and network reachability.

Service selectors and EndpointSlices can look like DNS failures

A Service can exist and resolve correctly while still sending traffic nowhere useful.

The Service selector determines which Pods belong behind the Service. Kubernetes represents the resulting backend addresses through EndpointSlices. If labels do not match, Pods are not ready, or ports are wrong, the DNS name may be perfectly healthy while the application remains unreachable.

Useful inspection commands include:

kubectl get svc <service> -n <namespace>
kubectl get endpointslice -n <namespace> \
  -l kubernetes.io/service-name=<service>

The key questions are:

• Does the Service exist in the namespace the caller expects?
• Does the selector match the intended Pods?
• Are endpoints present?
• Are those endpoints ready?
• Does targetPort match the port the application actually listens on?
• Is the client using the correct Service port?

This distinction matters because changing CoreDNS will not fix an empty EndpointSlice.

Likewise, if DNS returns the correct Service address but the application times out, focus on the traffic path after resolution. Check endpoints, readiness, network policies, and application listeners before modifying DNS configuration.

The related Kubernetes Services Explained: Ingress and Load Balancers guide owns the broader Service exposure model. This article stays on discovery and diagnosis.

CoreDNS health and Pod resolver settings narrow cluster-wide failures

CoreDNS is the standard DNS component used for Kubernetes Service discovery in current Kubernetes deployments. Even when CoreDNS is the implementation, the DNS Service and labels commonly retain the kube-dns name for compatibility.

When failures affect many Services or many Pods, inspect the cluster DNS components instead of debugging one application at a time.

Useful checks include:

kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns
kubectl get svc -n kube-system kube-dns
kubectl get endpointslice -n kube-system \
  -l kubernetes.io/service-name=kube-dns

These checks answer four separate questions:

1. Are the DNS Pods running?
2. Do their logs show resolution or forwarding errors?
3. Does the cluster DNS Service exist?
4. Does that Service have usable endpoints?

The Pod's own resolver policy also matters.

ClusterFirst is the normal cluster-first DNS behavior for Pods. Default makes a Pod inherit the node's resolver configuration. None requires explicit DNS settings through dnsConfig. Host-networked Pods need special consideration because their resolver behavior differs from ordinary Pods.

When only one workload has DNS problems, compare its Pod specification and /etc/resolv.conf with a working workload before changing cluster-wide CoreDNS configuration.

Kubernetes documentation also calls out node-level resolver issues as a possible source of CoreDNS forwarding loops, particularly when the node points at a local stub resolver rather than an upstream resolver file suitable for the cluster DNS configuration. That is an infrastructure-level diagnosis, not the first assumption for every failed lookup.

NetworkPolicy and upstream DNS can break resolution after CoreDNS is healthy

A healthy CoreDNS deployment does not guarantee every Pod can reach it.

When egress NetworkPolicy isolates a workload, DNS traffic must still be permitted to the cluster DNS path if the application needs name resolution. DNS can use both UDP and TCP, so restrictive policy should account for both protocols where port 53 access is intentionally controlled.

DNS policy should permit both UDP and TCP on port 53 when restrictive egress policy requires name resolution.

This creates a useful diagnostic split.

If a Pod receives an immediate NXDOMAIN, the DNS server responded but did not find the requested name. Investigate spelling, namespace, Service existence, and search domains.

If the query times out, the resolver may not be reachable. Investigate policy and network reachability to cluster DNS.

External names add another layer. CoreDNS normally forwards non-cluster queries according to its configured upstream resolution path. That means this pattern is especially informative:

kubernetes.default resolves
api.production resolves
example.com fails

When cluster names work but external names fail, the internal Service discovery path is functioning. Focus on CoreDNS forwarding, node/upstream resolver configuration, and any egress controls between cluster DNS and the upstream resolver.

The inverse pattern is also possible: external names may resolve through a custom Pod policy while cluster Service names fail. That points back toward the Pod's dnsPolicy, search domains, or cluster-DNS configuration.

For workload-level traffic isolation beyond DNS, use Kubernetes Network Policies for Small Teams.

The troubleshooting framework isolates the failure layer

Use this sequence during an incident before changing configuration.

The order is deliberate.

Do not restart CoreDNS because one Service name was misspelled. Do not relax NetworkPolicy because a Service selector is empty. Do not change Service exposure because one Pod uses the wrong DNS policy.

A good incident note should record the first layer that failed and the first layer that passed. That makes repeated DNS incidents easier to compare and prevents the team from rediscovering the same boundary every time.

NodeLocal DNSCache can be useful in some clusters to reduce DNS latency and connection-tracking pressure by running a local caching agent on nodes. It is an optimization architecture, not the first fix for an unresolved namespace, broken Service selector, or unhealthy CoreDNS deployment.

Raff keeps infrastructure networking separate from Kubernetes service discovery

Raff managed Kubernetes provides the cluster infrastructure boundary, including private VPC networking and the managed control-plane model. Kubernetes Service discovery remains a workload-layer concern: teams still define Service names, namespaces, selectors, DNS policies, application dependencies, and NetworkPolicies.

That separation is useful during incidents.

Raff VPC / cluster infrastructure
        ↓
Kubernetes cluster DNS
        ↓
Service identity
        ↓
EndpointSlices
        ↓
Application Pods

A private cluster network can be healthy while an application uses the wrong Service name. A Service can resolve while its EndpointSlice is empty. CoreDNS can be healthy while a restrictive egress policy blocks one namespace from reaching it.

Raff currently reports more than 15,000 VMs across the platform and publishes a 99.9% uptime SLA. Those platform signals do not turn every failed Service lookup into an infrastructure incident; the diagnosis still needs to identify which layer owns the failure.

For Raff teams, the practical review is therefore:

• verify the namespace and expected Service identity;
• verify the Service and its ready endpoints;
• verify the Pod resolver configuration;
• verify CoreDNS health only when evidence points there;
• verify NetworkPolicy and VPC-level reachability as separate layers;
• verify external upstream DNS only when cluster discovery already works.

This keeps remediation narrow. A workload configuration problem should be fixed as a workload configuration problem instead of widening infrastructure access or changing cluster-wide DNS unnecessarily.

Kubernetes DNS incidents are easier when names and ownership are explicit

Kubernetes DNS and Service discovery work best when applications depend on stable Service names rather than Pod addresses and when teams understand the namespace boundary behind those names.

When a lookup fails, identify the failure layer before changing infrastructure. Test the correct name, inspect the Pod resolver, verify the Service and EndpointSlices, confirm CoreDNS health, then inspect NetworkPolicy and upstream DNS only where the evidence points. If the name resolves but the application does not connect, continue the investigation after DNS rather than treating resolution as the complete network path.

For the broader traffic model, continue with Kubernetes Networking and Storage Architecture. For workload isolation, use Kubernetes Network Policies for Small Teams.

Sources

• Kubernetes DNS for Services and Pods
• Kubernetes Debugging DNS Resolution
• Kubernetes Using CoreDNS for Service Discovery
• Kubernetes Debug Services
• Kubernetes Network Policies