The Grafana Agent fails to reach pods via a Cluster IP service in Kubernetes on Ubuntu due to network policy restrictions, misconfigured endpoints, or DNS resolution issues. This guide resolves connectivity problems by verifying service definitions, checking pod readiness, and ensuring the agent’s namespace has proper permissions to access the target services within the cluster.
Verifying Service Endpoint Availability
Before troubleshooting complex networking layers, confirm that the Kubernetes service itself is healthy and has active endpoints. A Cluster IP is only assigned and routable if there are at least one running pod matching the service's selector labels. If the endpoint slice is empty, the service cannot route traffic regardless of the agent's configuration.
Use
kubectl get endpoints <service-name> to check for associated IPs. If the output shows an empty list, inspect the pods targeted by the service using
kubectl get pods -l app=<target-app>. Ensure these pods are in a
Running state and have passed their liveness probes. Often, the issue stems from the target pods crashing or failing health checks, leaving the service without valid destinations.
For a deeper dive into monitoring specific metrics, consider reviewing
View Per Core Statistics of Kubernetes Pods with mpstat, as understanding pod resource states can help diagnose why targets might be unreachable during high load scenarios.
Checking Network Policies and Namespace Isolation
Kubernetes NetworkPolicies can inadvertently block traffic between namespaces. By default, all ingress traffic is allowed, but if any NetworkPolicy exists in either the agent's namespace or the target service's namespace, it may restrict egress or ingress connections. The Grafana Agent typically runs in a dedicated monitoring namespace (e.g.,
monitoring) while targets reside in application namespaces (e.g.,
default or
production).
Review NetworkPolicies in both namespaces using
kubectl get networkpolicy --all-namespaces. Look for policies that deny all traffic by default or explicitly block ports used by your scraping targets (commonly 80, 443, or custom metric ports). If a policy blocks egress from the agent's namespace, you must add an egress rule allowing traffic to the target service's CIDR or port range. Conversely, ensure no ingress policy in the target namespace blocks incoming connections from the agent's service account or pod IP range.
Validating DNS Resolution and Service Discovery
Kubernetes uses CoreDNS to resolve service names to Cluster IPs. The Grafana Agent relies on standard Kubernetes service discovery mechanisms, which require correct DNS formatting. The service name must follow the pattern
<service-name>.<namespace>.svc.cluster.local or simply
<service-name>.<namespace> if the agent is in the same namespace.
Test DNS resolution from within the Grafana Agent pod by executing a debug shell:
kubectl exec -it <agent-pod> -- nslookup <service-name>.<namespace>. If the lookup fails or returns a non-existent domain, verify that the CoreDNS pods are running and healthy. Check the CoreDNS logs for errors related to zone transfers or upstream failures. Additionally, ensure that the service definition includes the correct protocol (
TCP or
UDP) in its port specification, as mismatches can sometimes cause silent failures in service discovery.
For related infrastructure challenges, see
Kubernetes RBAC: Troubleshooting Access to Resources, as improper RBAC settings can also prevent the agent from listing endpoints or reading service metadata required for dynamic scraping.
Inspecting Grafana Agent Configuration
The Grafana Agent's configuration file (
agent-config.yaml) defines how it discovers and scrapes targets. Incorrectly configured
kubernetes_sd_configs can lead to missed targets. Ensure the role is set to
endpoints or
service depending on your needs. If using
endpoints, the agent will scrape individual pod IPs; if using
service, it scrapes the Cluster IP.
Verify that the authorization section allows access to the Kubernetes API server. The agent requires a ServiceAccount with permissions to read pods, services, and endpoints. Check the
authorization block in the config to ensure the bearer token is correctly mounted or specified. Furthermore, inspect the
relabel_configs to ensure no rules are dropping the target labels before scraping begins. Misconfigured relabeling is a common cause where the agent finds the service but filters out all discovered instances.
Testing Connectivity with Debug Tools
If the above steps do not resolve the issue, perform manual connectivity tests from the agent pod to the target service. Use
curl or
wget inside the agent pod to hit the Cluster IP directly:
kubectl exec -it <agent-pod> -- curl -v http://<cluster-ip>:<port>. A connection timeout indicates a network-level block, such as a firewall or NetworkPolicy. A refusal or reset suggests the target service is not listening or rejecting connections.
If
curl succeeds but the agent still fails, the issue likely lies in the agent's internal logic or authentication. Check the agent's logs for specific error messages related to HTTP status codes (e.g., 401 Unauthorized, 500 Internal Server Error). These logs provide precise clues about whether the problem is network-related or application-level. For persistent connectivity issues across different environments, exploring
WireGuard Client Works Perfectly with Docker and Kubernetes might offer insights into advanced networking setups that could interfere with standard service mesh behaviors.
Frequently Asked Questions
Why does the Grafana Agent show targets as "down" in Prometheus?
Targets appear down if the agent cannot establish a successful HTTP connection to the scrape endpoint. This often results from incorrect port mappings in the service definition or the target application not exposing metrics on the expected path. Verify the scrape path matches the application's exposed metrics endpoint, typically
/metrics.
Can I use NodePort instead of Cluster IP for the Grafana Agent?
While possible, using NodePort exposes services on every node's IP, which increases security risks and complicates service discovery. Cluster IP is preferred for internal communication because it remains stable within the cluster and is easier to manage with NetworkPolicies. Only use NodePort if external access is strictly required and properly secured.
How do I troubleshoot DNS issues in Kubernetes?
Start by checking the CoreDNS deployment status and logs. Ensure that the
coredns pods are running and not crashing. Test resolution from various pods to determine if the issue is widespread or isolated to specific namespaces. Clearing the CoreDNS cache or restarting the pods can sometimes resolve transient DNS propagation delays.
What permissions does the Grafana Agent ServiceAccount need?
The ServiceAccount requires read access to pods, services, endpoints, and nodes within the monitored namespaces. It typically needs the
view role or a custom RoleBinding that grants
get,
list, and
watch permissions on these resources. Without these permissions, the agent cannot discover targets dynamically.
Does the Grafana Agent support HTTPS scraping?
Yes, the Grafana Agent supports HTTPS scraping by configuring the
tls_config in the scrape job definition. You must provide the appropriate CA certificate, client certificate, and key if mutual TLS is required. Ensure the target service presents a valid certificate that the agent trusts, or disable certificate verification for testing purposes only.