Unreachable endpoints on specific nodes in a Cilium Kubernetes cluster usually stem from node-specific network policy enforcement, BGP peering failures, or local kube-proxy conflicts. To resolve this, verify the health of the Cilium agent on the affected node, check for overlapping NetworkPolicies blocking traffic, and ensure that the underlying CNI interface is correctly configured without interference from other networking tools.Verifying Node-Specific Agent Health
The first step in troubleshooting unreachable endpoints on a subset of nodes is to determine if the Cilium agent itself is functioning correctly on those specific machines. Unlike global cluster issues, node-specific problems often indicate that the agent pod on the target node has crashed, entered a CrashLoopBackOff state, or failed to initialize its datapath. You should start by checking the status of the Cilium pods scheduled on the affected nodes using standard Kubernetes diagnostic commands.
kubectl get pods -n kube-system -l k8s-app=cilium -o wide | grep <affected-node-name>If the agent pod is not running, inspect its logs for initialization errors. Common issues include failure to load kernel modules, inability to mount necessary filesystems like /sys/fs/bpf, or permission denied errors when accessing the host network namespace. If the agent is running but reports unhealthy status, use the Cilium CLI to get a detailed status report which provides insights into the connectivity and tunneling status.
cilium status --waitLook specifically for errors related to "Datapath" or "CNI" configuration. If the agent is healthy but endpoints are still unreachable, the issue likely lies in the data plane configuration rather than the control plane agent itself. This distinction is crucial because it directs your troubleshooting efforts toward network policies and interface configurations instead of general cluster health.
Investigating Overlapping Network Policies
Cilium’s strength lies in its identity-based security model, but this also means that misconfigured NetworkPolicies can silently drop traffic to specific endpoints. If only certain nodes have unreachable endpoints, it is possible that a NetworkPolicy with a node selector or endpoint selector is inadvertently blocking traffic destined for workloads on those nodes. Review all active NetworkPolicies in the namespaces where the unreachable endpoints reside.
Pay close attention to policies that use nodeSelector or match labels present on the affected nodes. A common mistake is creating a policy that allows ingress from a specific CIDR but fails to account for the internal IP ranges used by the cluster on specific availability zones or node pools. Use the Cilium CLI to debug connectivity between two pods, which will show exactly where the packet is being dropped.
cilium connectivity test --namespace <target-namespace>This command creates a set of test pods and attempts to establish connections between them across different nodes. The output will clearly indicate which connection failed and provide the reason code, such as "Denied by policy" or "No route to host." If the reason is a policy denial, examine the policy definitions to see if they need adjustment to allow traffic from the source nodes. Remember that Cilium policies are cumulative; if any policy denies traffic, it is blocked, even if other policies allow it.
Checking for Kube-Proxy Conflicts and Interface Issues
In many Kubernetes distributions, both kube-proxy and Cilium operate simultaneously during migration periods or default installations. Kube-proxy uses iptables or ipvs to manage service routing, which can conflict with Cilium’s eBPF-based routing. If kube-proxy is enabled on the affected nodes, it may interfere with Cilium’s ability to correctly route traffic to endpoints, leading to intermittent or complete unreachability.
To resolve this, ensure that kube-proxy is disabled on nodes where Cilium is installed. You can do this by setting the --proxy-mode=off flag in the kube-proxy DaemonSet manifest or by deleting the kube-proxy pods entirely if Cilium is intended to be the sole CNI. Additionally, check for any custom network interfaces or VLAN configurations on the affected nodes that might not be recognized by Cilium’s interface discovery mechanism.
Sometimes, the issue is not software-related but physical or virtual infrastructure-related. For instance, if you are running Kubernetes on VMs, check the hypervisor settings to ensure that promiscuous mode is enabled if required by your network setup, although modern eBPF setups often do not require this. Compare the network interface names and MTU settings between working and non-working nodes. Any discrepancy in MTU can lead to fragmented packets that are dropped by firewalls or routers in the path.
Comparing with Related Networking Issues
While this issue is specific to Cilium, similar symptoms can appear in other networking contexts. For example, if you are managing Ubuntu VMs on KVM, you might encounter internet connection issues due to incorrect bridge configurations, which is distinct from intra-cluster endpoint reachability but shares the root cause of interface misconfiguration. Similarly, troubleshooting DNS resolution failures in Kali Linux or other environments often involves checking whether the resolver is correctly pointing to the cluster’s CoreDNS service, which Cilium heavily relies on for service discovery.
If you suspect broader network connectivity issues beyond the cluster, such as inability to reach external services, you might find relevant guidance in resources discussing temporary failure in name resolution or general internet connectivity problems. However, for internal endpoint reachability, focus strictly on the Cilium agent and policy layer first.
Reviewing BGP and External Connectivity
If your cluster uses BGP to advertise services to the external network via Cilium’s LoadBalancer functionality, unreachable endpoints might be due to BGP session instability on specific nodes. Check the BGP peer status on the affected nodes using the Cilium CLI. A downed BGP session will prevent external traffic from reaching the LoadBalancer IPs assigned to services on those nodes.
cilium bgp peersEnsure that the BGP router configuration matches the expected ASN and peer IP addresses. Misconfigurations here often affect only a subset of nodes if the BGP speaker is not properly initialized on those specific machines. Restarting the Cilium agent on the affected nodes can sometimes re-establish the BGP session if it was stuck in a flapping state.
Conclusion
Resolving unreachable endpoints on specific nodes in a Cilium cluster requires a systematic approach: verify agent health, audit network policies for unintended blocks, eliminate kube-proxy conflicts, and check infrastructure-level settings like MTU and BGP sessions. By isolating the problem to the specific node and using Cilium’s built-in debugging tools, you can quickly identify and fix the root cause.
Frequently Asked Questions
Why does cilium connectivity test show successful tests but real applications fail?
This discrepancy often occurs when the connectivity test uses different source or destination labels than your actual application pods. Ensure that the test pods are deployed in the same namespaces and have matching labels to the failing application pods to accurately replicate the traffic flow.
Can I disable kube-proxy completely in a Cilium cluster?
Yes, Cilium is designed to replace kube-proxy for service routing. Disabling it reduces resource overhead and prevents iptables conflicts. However, you must ensure that Cilium is fully operational and that no legacy dependencies rely on kube-proxy for core cluster functions before removal.
What should I do if the Cilium agent crashes repeatedly on a node?
Check the system logs on the node for OOM kills or disk space issues, as these are common causes of agent crashes. Also, verify that the kernel version supports the required eBPF features and that there are no conflicting kernel modules loaded that might interfere with the Cilium datapath.
How do I check if a NetworkPolicy is blocking my traffic?
Use the cilium endpoint get command to view the effective policy IDs applied to an endpoint. Then, use cilium policy list to review the details of those policies. The connectivity test tool also provides explicit reasons for drops, making it easier to pinpoint the denying rule.
Does MTU mismatch always cause unreachable endpoints?
Not always, but it frequently causes silent packet loss for larger payloads. If small pings succeed but large transfers fail, check the MTU settings on the network interfaces of the affected nodes and ensure they match the cluster-wide MTU configuration, typically 1450 or 1480 bytes depending on overlay encapsulation.