Kubernetes pods often lose internet access due to misconfigured CNI plugins, incorrect DNS settings, or restrictive network policies. This guide walks you through the essential diagnostic steps to restore connectivity by verifying node-level networking, inspecting pod DNS resolution, and auditing firewall rules on your cluster nodes.
Verify Node-Level Connectivity
Before diving into Kubernetes-specific configurations, you must ensure that the underlying infrastructure can reach the internet. The most common cause of pod isolation is a failure at the host level rather than within the cluster itself. If the worker nodes cannot route traffic externally, no amount of Kubernetes configuration will fix the issue for the pods scheduled on those nodes.
Start by accessing one of the affected worker nodes via SSH. You should attempt a basic ping to an external IP address, such as
8.8.8.8, to test raw network reachability. Simultaneously, try resolving a domain name using
nslookup google.com or
dig google.com. If these commands fail, the problem lies with the node’s network interface, routing table, or local firewall rules (such as iptables or firewalld). Check the node’s default gateway using
ip route to ensure it points to the correct upstream router. In cloud environments, verify that the security groups or network ACLs associated with the instance allow outbound traffic on ports 53 (DNS) and 443/80 (HTTP/HTTPS).
If node-level connectivity is confirmed, proceed to check if the issue is isolated to specific nodes or affects the entire cluster. Run the connectivity tests on multiple nodes to rule out hardware or network switch failures affecting a single data path.
Inspect Cluster Network Configuration
Once node connectivity is established, the next step is to examine how Kubernetes handles pod networking. Most clusters rely on a Container Network Interface (CNI) plugin like Calico, Flannel, Cilium, or Weave Net. A broken CNI installation or a version mismatch between the CNI and the Kubernetes version is a frequent culprit for silent packet drops.
Check the status of your CNI pods running in the
kube-system namespace. Use
kubectl get pods -n kube-system and look for components like
calico-node or
flannel. Ensure they are in a Running state and not restarting frequently. If a CNI pod is crashing, check its logs with
kubectl logs <pod-name> -n kube-system for errors related to IP allocation or interface creation.
Additionally, verify that the CNI has successfully allocated IP addresses to your pods. Describe a problematic pod using
kubectl describe pod <pod-name> and look at the events section. If you see errors about failing to allocate an IP or creating network interfaces, the CNI plugin may be misconfigured. Ensure that the pod’s network namespace is correctly linked to the host’s bridge or overlay network. For example, in Flannel-based clusters, check that the
cni0 bridge exists on the node and has the correct subnet configured.
Diagnose DNS Resolution Issues
Even with perfect network routing, pods cannot access the internet if they cannot resolve domain names. Kubernetes uses CoreDNS as the default DNS provider, and misconfigurations here often manifest as timeouts when trying to reach external services.
To test DNS from within a pod, exec into a busybox or debug container and run
nslookup kubernetes.default followed by
nslookup google.com. If the first command succeeds but the second fails, the issue is likely with external DNS forwarding. Check the
CoreDNS deployment in the
kube-system namespace. Inspect its ConfigMap to ensure the
forward directive points to valid upstream DNS servers, such as
8.8.8.8 or
1.1.1.1.
If both lookups fail, the CoreDNS service itself might be unreachable or crashed. Verify that the
kube-dns or
coredns service has an active endpoint. Sometimes, network policies inadvertently block traffic to the DNS service on port 53 UDP/TCP. Review any applied NetworkPolicies to ensure they allow egress from pods to the
kube-system namespace on port 53.
Audit Network Policies and Firewalls
NetworkPolicies are powerful tools for restricting traffic, but they are also a common source of accidental lockouts. By default, if no NetworkPolicy is defined, all traffic is allowed. However, if you have applied even a single policy, it may restrict egress traffic to the internet.
List all NetworkPolicies in your namespaces using
kubectl get networkpolicies --all-namespaces. Examine each policy to see if it includes an
egress rule. A restrictive egress rule might only allow traffic to internal services, blocking all external HTTP/HTTPS requests. To test if a policy is the cause, temporarily remove the offending policy using
kubectl delete networkpolicy <name> and retry the internet access test. If connectivity returns, you need to update the policy to explicitly allow egress to
0.0.0.0/0 on ports 80 and 443.
Similarly, check for global network policies or third-party firewall solutions integrated with your cloud provider. Some enterprise Kubernetes platforms enforce strict egress filtering at the cluster level. Consult your platform documentation to identify if there are global deny-all rules that override individual pod configurations.
Test with a Minimal Pod
When troubleshooting complex networking issues, isolating variables is crucial. Create a minimal pod using a simple image like
busybox to eliminate application-specific code as a variable. Deploy this pod without any custom labels or selectors to ensure it is not targeted by existing NetworkPolicies.
Run the following command to create a test pod:
kubectl run test-internet --image=busybox --command -- sleep 3600
Once the pod is running, exec into it and perform the connectivity tests again. Ping
8.8.8.8 and try to download a file using
wget or
curl. If this minimal pod works while others do not, the issue is likely related to specific labels, namespaces, or resource limits assigned to the failing workloads. If the minimal pod also fails, the problem is systemic to the cluster’s network plane.
For more insights on general networking diagnostics, you might find our guide on
Ping Strange Behavior: Troubleshooting Networking and DNS Issues helpful for understanding packet loss patterns. Additionally, if you suspect broader infrastructure issues, reviewing
Troubleshooting Website Access Between Two Offices on Different Subnets can provide techniques for diagnosing cross-subnet routing problems that may apply to your node-to-internet path.
Frequently Asked Questions
Why can I ping an IP but not a domain name?
This indicates that network routing is working, but DNS resolution is failing. Check your CoreDNS configuration and ensure the forward directive points to reliable upstream servers. Also, verify that no NetworkPolicy is blocking UDP port 53 traffic from the pod.
How do I check if a CNI plugin is installed correctly?
You can verify the CNI status by listing pods in the
kube-system namespace. Look for components like
calico-node or
flannel and ensure they are running without restarts. Check their logs for errors related to IP allocation or interface binding.
Can a NetworkPolicy block internet access?
Yes, if a NetworkPolicy defines an egress rule, it denies all traffic not explicitly allowed. Ensure your policy includes an egress entry for ports 80 and 443 to destinations like
0.0.0.0/0. Removing the policy temporarily can help confirm if it is the cause.
What should I do if all pods on a node lose connectivity?
If all pods on a specific node are affected, the issue is likely at the node level. Check the node’s network interface, routing table, and local firewall rules. Test connectivity directly from the node’s shell to determine if the host itself can reach the internet.
Is it safe to disable NetworkPolicies during troubleshooting?
It is generally safe to temporarily remove restrictive NetworkPolicies for debugging purposes, provided your environment is not exposed to untrusted public traffic. Always reapply the necessary policies immediately after identifying the root cause to maintain security.