The Prometheus Stack Operator fails to scrape cluster application metrics when ServiceMonitor resources are misconfigured, network policies block traffic, or the target pods lack proper annotations. This issue prevents visibility into application performance and can lead to silent failures in production environments. Resolving this requires verifying the ServiceMonitor selector matches the target Service labels, ensuring the correct port name is defined in the Service spec, and confirming that the Prometheus server has network access to the pod endpoints.
Verifying ServiceMonitor Configuration
The most common cause for scraping failures is a mismatch between the ServiceMonitor resource and the Kubernetes Service it attempts to monitor. The ServiceMonitor acts as a bridge, telling the Prometheus operator which services to watch and how to reach them. If the selector does not match the labels on the Service, or if the port name is incorrect, Prometheus will silently ignore the target.
To diagnose this, first check the ServiceMonitor definition using
kubectl get servicemonitor -n monitoring. Examine the
spec.selector.matchLabels field. These labels must exactly match the labels applied to the Kubernetes Service you intend to monitor. For example, if your Service has the label
app: my-app, the ServiceMonitor selector must include
app: my-app. A single typo here will result in zero targets being discovered.
Next, inspect the
spec.endpoints section of the ServiceMonitor. Ensure the
port field references a named port in the Service definition. Kubernetes Services expose ports by name, not just by number. If the Service defines a port named
http-metrics, the ServiceMonitor endpoint must specify
port: http-metrics. Using the numeric port value directly in the ServiceMonitor is not supported in standard configurations and will lead to discovery errors.
You can also verify the Service itself with
kubectl get service -n <namespace>. Look at the
PORT(S) column. Confirm that the port name listed there matches what you expect. If the Service exposes port 8080 but names it
web, your ServiceMonitor must use
port: web. This naming convention is strict and does not allow for ambiguity.
Checking Network Policies and Pod Accessibility
Even with perfect configuration, network restrictions can prevent the Prometheus server from reaching the application pods. In modern Kubernetes clusters, NetworkPolicies are often enabled by default to restrict traffic flow. If a NetworkPolicy exists in the namespace where your application runs, it might be blocking incoming connections from the Prometheus namespace.
Review any NetworkPolicies in the application namespace using
kubectl get networkpolicy -n <application-namespace>. Check the
ingress rules to see if they allow traffic from the Prometheus server pods. The Prometheus server typically runs in a dedicated namespace like
monitoring or
prometheus. You may need to add an ingress rule that allows traffic from pods with the label
app.kubernetes.io/name: prometheus in the monitoring namespace.
If no explicit NetworkPolicy is found, consider the default behavior. Some clusters have a default deny-all policy for ingress traffic. In such cases, all incoming traffic is blocked unless explicitly allowed. You can test connectivity by port-forwarding from your local machine to the Prometheus server and checking the targets page. If the targets show as "Down" with a connection timeout error, it strongly suggests a network-level block rather than a configuration error.
For a deeper dive into repairing Prometheus configurations, you might find
The Prometheus Repair Process: A Comprehensive Guide helpful for understanding broader system integrity checks.
Validating Target Discovery via Prometheus UI
The Prometheus UI provides the most direct evidence of what is happening during the scraping process. Navigate to the Prometheus admin interface, usually accessible via port-forwarding or an Ingress resource. Go to the
Status menu and select
Targets. This page lists all discovered endpoints and their current health status.
Look for your application’s targets. They should appear under the job name defined in your ServiceMonitor. If the targets are missing entirely, the issue lies in service discovery, meaning the ServiceMonitor selector or Service labels are mismatched. If the targets appear but show a state of
DOWN, examine the last error message displayed. Common errors include
context deadline exceeded (timeout) or
connection refused (service not listening).
A timeout often indicates that the Prometheus server cannot reach the pod IP, pointing back to network policies or DNS issues. A connection refused error means the pod is reachable, but the application is not listening on the specified port or the port name is wrong. Click on the individual target instance to see detailed scrape logs. This will show the exact HTTP request made and the response received, helping you determine if the application is returning valid metrics in the expected format.
If you are dealing with complex routing issues involving external access, reviewing
Kubectl Ingress: A Comprehensive Guide for Tech Support can help ensure your ingress controllers are not interfering with internal service discovery.
Ensuring Correct Annotations on Application Pods
While less common with the Stack Operator, which relies heavily on ServiceMonitors, some setups require specific annotations on the application pods themselves. The Prometheus operator uses these annotations to identify metric endpoints if automatic discovery is not fully configured. Check the pod template in your Deployment or StatefulSet.
Look for annotations starting with
prometheus.io/. Specifically, check for
prometheus.io/scrape, which should be set to
true, and
prometheus.io/port, which should match the container port. However, note that the Stack Operator primarily uses ServiceMonitors for service-based discovery. If you are using ServiceMonitors, these pod annotations are often redundant. But if you have switched to pod-level discovery, ensure these annotations are present and correctly formatted.
Verify that the application container is actually exposing metrics. Many applications only expose metrics on specific paths, such as
/metrics or
/health. If the application serves metrics on a different path, you must configure the
path field in the ServiceMonitor endpoint. The default path is usually
/metrics, but custom applications may vary.
Conclusion
Resolving Prometheus scraping issues involves a systematic approach: verify label matching, confirm port names, check network policies, and inspect the Prometheus UI for error details. By carefully aligning the ServiceMonitor configuration with the Kubernetes Service and ensuring network accessibility, you can restore full visibility into your cluster applications. Regularly checking the Targets page helps catch these issues early before they impact alerting and dashboards.
Frequently Asked Questions
Why do my Prometheus targets show as Down instead of Up?
Targets show as Down when the Prometheus server cannot successfully scrape the endpoint within the configured timeout period. This is often caused by network policies blocking traffic, incorrect port numbers, or the application failing to respond to health checks. Check the last error message in the Prometheus UI for specific clues like timeouts or connection refusals.
How do I find the correct port name for my ServiceMonitor?
Run
kubectl get service <service-name> -n <namespace> and look at the
PORT(S) column. The name listed after the port number (e.g.,
8080:TCP/http-metrics) is the name you must use in the ServiceMonitor's
port field. Do not use the numeric port value directly in the ServiceMonitor configuration.
Can I use Prometheus without the Stack Operator?
Yes, you can run vanilla Prometheus outside of the Stack Operator, but you lose the automated ServiceMonitor reconciliation. Without the operator, you must manually manage Prometheus configurations and reloads. The Stack Operator simplifies this by watching Kubernetes resources and updating the Prometheus config automatically.
What causes a context deadline exceeded error in Prometheus?
This error typically indicates a network timeout, meaning the Prometheus server could not establish a connection to the target pod within the scrape interval. It often points to restrictive NetworkPolicies, DNS resolution failures, or the target pod being stuck in a CrashLoopBackOff state. Verify that the target pod is running and accepting connections.
How can I test if my application is exposing metrics correctly?
Use
kubectl port-forward <pod-name> <local-port>:<container-port> to forward traffic locally. Then, use
curl localhost:<local-port>/metrics to see if the metrics output is returned in the correct Prometheus text format. If curl returns HTML or an error, the application is not serving metrics at that path.