Incorrect network metrics in Grafana, Prometheus, InfluxDB, and Telegraf usually stem from misconfigured interfaces, unit mismatches, or counter resets. Verify the interface selection in Telegraf, ensure consistent byte versus bit units across your stack, and check for system reboots that reset counters to accurately diagnose and fix these data discrepancies.
Verifying Interface Selection in Telegraf
The most common cause of unexpected network metric values is selecting the wrong network interface in the Telegraf configuration file. Telegraf collects data based on theinterfaces list defined in its plugin configuration. If you are monitoring a specific server but the configuration points to a virtual interface like docker0 or lo (loopback), the metrics will not reflect the actual physical network traffic. You must identify the correct interface name using command-line tools such as ip link show or ifconfig before updating the configuration.
Once the correct interface name is identified, open the telegraf.conf file located in /etc/telegraf/. Locate the [inputs.net] section and update the interfaces array to include only the relevant physical adapters. For example, if your primary interface is eth0, the configuration should look like this:
[inputs.net]
## By default stats are gathered from all network interfaces,
## specifying an interface will gather stats just from that interface.
interfaces = ["eth0"]
systemctl restart telegraf. It is crucial to allow a few minutes for new data to be ingested into your backend database. If you continue to see zero or negligible values, double-check that the interface name matches exactly, including case sensitivity, as Linux systems distinguish between Eth0 and eth0.
Addressing Unit Mismatches Between Systems
Even when the correct interface is selected, values may appear incorrect due to unit conversions between Prometheus, InfluxDB, and Grafana. Network metrics are often reported in bytes per second by collectors like Telegraf, but dashboards frequently expect bits per second (bps) or kilobits per second (Kbps). Prometheus stores raw samples, while InfluxDB might apply different scaling factors depending on how the data was written. In Grafana, check the query editor for your network panel. Ensure that the unit conversion is applied correctly at the visualization level rather than relying on the underlying data source to handle it. If Prometheus is receiving data from Telegraf, verify that the metric names are consistent. For instance,net_bytes_recv should be clearly distinguished from net_bits_recv. A common mistake is summing bytes without converting them to bits when comparing against thresholds set in bps.
To resolve this, you can use PromQL functions to scale the data. Multiply the value by 8 to convert bytes to bits:
rate(node_network_receive_bytes_total[5m]) * 8If you are using InfluxDB, ensure that the continuous queries or retention policies are not altering the precision of the float values. Misaligned decimal places can make high-speed network traffic appear as low-value noise. Always standardize on one unit (preferably bits per second for networking) across the entire pipeline from collection to visualization.Handling Counter Resets and System Reboots
Network interfaces often expose counters that reset upon system reboot or interface down/up events. Prometheus uses `rate()` or `irate()` functions to calculate the per-second average over a time window. However, if a counter resets to zero during the evaluation window, the rate calculation may result in negative values or spikes that distort the graph. This is particularly problematic in environments with frequent deployments or maintenance windows. To mitigate this, ensure that your dashboard panels filter out negative rates. In Grafana, you can add a condition to hide negative values or clamp them to zero. Additionally, check the uptime of the monitored hosts. If the host has recently restarted, the initial few minutes of data may be inaccurate until the counters stabilize. For InfluxDB, similar issues arise if the measurement does not account for wraparounds. While InfluxDB does not automatically handle counter resets like Prometheus's `rate()`, you can use `DERIVATIVE()` or custom scripts to detect jumps and adjust accordingly. It is best practice to monitor the `up` metric alongside network metrics to correlate drops in traffic with known downtime events.Checking Data Retention and Query Time Ranges
Sometimes the issue is not with the data itself but with how it is queried. Incorrect time ranges in Grafana panels can lead to misleading averages. If you select a short time range (e.g., last 1 hour) for a metric that experiences bursty traffic, the average may appear artificially low. Conversely, a very long time range might smooth out critical anomalies. Ensure that the time picker in Grafana aligns with the resolution of your data. High-resolution data (e.g., 15-second intervals) should be viewed in shorter time windows to preserve detail. If you are seeing gaps in the data, check the health of the Telegraf agent and the connectivity to Prometheus or InfluxDB. Network latency or packet loss between the collector and the storage backend can cause missing data points, which Grafana may interpolate incorrectly. Finally, review the logs of each component. Telegraf logs can reveal dropped measurements, Prometheus logs can indicate scrape failures, and InfluxDB logs can show write errors. Correlating these logs with the timeline of incorrect values will often pinpoint the exact moment the discrepancy began.Frequently Asked Questions
Why do my network graphs show negative values?
Negative values typically occur when a counter resets to zero during the calculation window, causing the difference between two points to be negative. To fix this, configure your Grafana panel to hide negative values or use a mathematical function to clamp them to zero.How can I tell if Telegraf is collecting from the wrong interface?
You can verify the active interface by running `ip link show` on the host and comparing the output to the `interfaces` list in `telegraf.conf`. If the configured interface name does not match the physical adapter, update the configuration and restart the service.Should I use bits or bytes for network metrics?
Bits per second is the industry standard for networking speeds, but consistency is key. Choose one unit and apply the appropriate conversion factor (multiplying bytes by 8) in your query language or dashboard settings to ensure accurate comparisons.What causes gaps in network metric data?
Gaps are usually caused by Telegraf agent crashes, network connectivity issues between the collector and database, or firewall blocks preventing data transmission. Check the application logs of both Telegraf and your backend database for error messages related to connection timeouts or write failures.Does a system reboot affect Prometheus network metrics?
Yes, because Prometheus calculates rates based on counter values, a reboot resets the counters to zero. This can create artificial spikes or dips in the graph immediately after the restart. Monitor the host's uptime metric to identify when these resets occur and exclude those periods from analysis if necessary.