Prometheus uses the Pushgateway to collect metrics from short-lived jobs that cannot be scraped directly. When managing multiple exporters behind a push-proxy architecture, you must configure each job with unique labels to prevent metric collisions. This setup ensures that data from distinct sources remains isolated and accurately represented in your time-series database without overwriting previous values.
Understanding the Pushgateway Role
The Pushgateway is not a proxy in the traditional network sense but an intermediary storage unit for metrics. It allows ephemeral processes, such as batch scripts or CI/CD pipelines, to send their monitoring data to Prometheus. Without this component, these transient jobs would disappear before Prometheus could scrape them during its regular cycle. The gateway stores these metrics until they are explicitly deleted or expire based on TTL settings.
When you deploy several exporters that use the push model, you are essentially creating a hub-and-spoke topology. Each exporter pushes its data to the central Pushgateway. Prometheus then scrapes this single endpoint rather than trying to reach every individual short-lived instance. This approach simplifies network configuration because you only need one open port for scraping, regardless of how many internal jobs are running.
It is crucial to distinguish between the standard pull-based scraping and this push-based mechanism. Pull-based scraping is preferred for long-running services like web servers or databases. However, for tasks that last only seconds or minutes, pushing is the only viable option. The Pushgateway aggregates these pushes into a unified view for Prometheus to consume.
Configuring Unique Labels for Multiple Exporters
The most critical aspect of using multiple exporters with the Pushgateway is label management. If two different jobs push metrics with identical label sets, the second job will overwrite the first. This leads to data loss and incorrect alerting. To avoid this, you must assign unique identifiers to each exporter instance.
You should include at least two labels in every push:
instance and
job. The
job label typically identifies the type of service, while the
instance label distinguishes between different runs or hosts. For example, if you have a backup script and a cleanup script, they might share the same
job name but require different
instance values.
Here is how you structure the push command for a specific exporter:
curl -X POST \
http://pushgateway:9091/metrics/job/my_job/instance/my_instance \
--data-binary @metrics.txt
In this URL structure,
my_job and
my_instance are path parameters. You must ensure that no other active job uses this exact combination. If you are running multiple instances of the same exporter on different servers, each server must have a unique
instance value. This prevents the metrics from colliding when they are aggregated by Prometheus.
Handling Metric Groups
The Pushgateway organizes metrics into groups defined by the label set. A group is uniquely identified by all its labels. When you push data, you are either adding to an existing group or creating a new one. If you push to a group that already exists, the new metrics replace the old ones entirely. This behavior is intentional but can be surprising if you expect additive merging.
For multiple exporters, this means you should design your labeling scheme so that each logical source has its own group. Do not rely on the gateway to merge metrics from different sources automatically. Instead, treat each exporter’s output as a separate entity within the gateway. This clarity helps when debugging why certain metrics appear missing or duplicated in your dashboards.
If you need to clean up old data, you can delete a specific group by sending a DELETE request to the same URL used for pushing. This is useful for ensuring that stale metrics from completed jobs do not clutter your queries.
Prometheus Configuration for Scraping
Once your exporters are pushing data correctly, you need to configure Prometheus to scrape the Pushgateway. This configuration is straightforward because there is only one target to monitor. You define a static job in your
prometheus.yml file that points to the Pushgateway’s address.
scrape_configs:
- job_name: 'pushgateway'
static_configs:
- targets: ['pushgateway:9091']
This basic configuration tells Prometheus to fetch all currently stored metrics from the gateway every 15 seconds (or whatever interval you set). Note that Prometheus does not scrape the individual exporters directly in this setup. It only sees the aggregated view provided by the gateway.
You may also want to adjust the scrape interval or add relabeling rules if you have a complex environment. For instance, if you want to filter out specific jobs from being scraped, you can use
metric_relabel_configs. However, for most multi-exporter setups, the default configuration is sufficient. The key is ensuring that the gateway is reachable from the Prometheus server and that the firewall rules allow traffic on port 9091.
Best Practices for Stability
To maintain a stable monitoring system, you should implement health checks for both the exporters and the Pushgateway itself. If the gateway goes down, all pushed metrics are lost unless they are persisted. Therefore, consider deploying the Pushgateway in a highly available manner or using persistent storage if supported by your version.
Additionally, monitor the size of the metric groups. If you fail to delete completed jobs, the gateway’s memory usage will grow indefinitely. Implement a cron job or a script that cleans up metrics after a successful completion. This proactive maintenance prevents performance degradation and ensures that your Prometheus queries remain fast and accurate.
For more advanced visualization of these metrics, you might explore how to structure your Grafana dashboards effectively. Understanding how to separate variables for different panels can help you isolate data from various exporters clearly. You can read more about this optimization here:
Grafana Dashboard: Separate Variables for Different Panels.
Finally, always test your push commands manually before automating them. Use tools like
curl to verify that the metrics appear in the Pushgateway’s UI and are subsequently scraped by Prometheus. This validation step catches labeling errors early, saving time during production deployments.
Frequently Asked Questions
What happens if I don't use unique labels?
Metrics from different jobs will overwrite each other because the Pushgateway treats identical label sets as the same data group. This results in data loss where only the last pushed value is retained. Always ensure your
instance and
job labels are distinct for every exporter.
Can I use Pushgateway for long-running services?
No, it is designed for short-lived jobs. Long-running services should use direct scraping via HTTP endpoints. Using Pushgateway for persistent services adds unnecessary latency and complexity to your architecture. Stick to push-only for ephemeral tasks.
How do I delete metrics from the Pushgateway?
Send an HTTP DELETE request to the specific metric group URL. For example,
curl -X DELETE http://pushgateway:9091/metrics/job/my_job/instance/my_instance. This removes the entire group associated with those labels immediately.
Does Pushgateway persist data across restarts?
By default, no. Data is stored in memory and is lost when the container or process stops. You can configure persistence using volume mounts if your deployment supports it, but this is not enabled out of the box. Plan your cleanup strategies accordingly.
Why are my metrics missing in Prometheus?
Check if the Pushgateway is actually scraping the correct job name. Also, verify that the labels in your push URL match exactly what Prometheus expects. Typos in the
job or
instance paths are common causes for missing data.