6

I'm using a PromQL query to calculate the cumulative traffic pushed/received through some interfaces on any node on the last 60 minutes. With Prometheus Node Exporter's metrics:

delta(node_network_receive_bytes_total{device=~"ens.*"}[60m])*8

And it's perfectly fine as far as the node does not reboot in that interval, the value is simply the difference between the vector's tip and its tail. When the system reboots and the counter resets, the meaning of the function stops reflecting said result.

e.g. Being this the graph for node_network_transmit_bytes_total:

IPv6 traffic

... the function will return -9MiB, instead of 10.2MiB.

I guess I can play out with rate()s to get an estimate by also using the time. But is there a better function/way to get the actual thing?

1 Answer 1

10

As indicated in the documentation of delta():

delta should only be used with gauges.

You should be using the increase() function which is specific to counters.

Breaks in monotonicity (such as counter resets due to target restarts) are automatically adjusted for.

This is one of the main reason to distinguish between gauge and counters. See this answer about the difference.

You can identify counters by one of the following methods:

  • the type in the textfile output (ex: # TYPE http_requests_total counter)
  • the value is monotonically increasing (grafana proposes counter related function when it detects it)
  • the name should end with _total (if exporter respects best practices)
3
  • This is exactly it! Next time I should try and read the docs properly... Thanks a bunch!
    – Alfageme
    Commented Jan 9, 2020 at 0:17
  • 2
    A gauge can be monotonically increasing. To distinguish for a gauge you care about the absolute value, for a counter how fast it is increasing. E.g. commits in a git repo is (usually) monotonically increasing, but it's a gauge rather than a counter. Commented Jan 9, 2020 at 7:56
  • @brian-brazil Right. Someone may reset, squash and forcibly push commits sometimes, so commits may decrease.
    – Lewis Chan
    Commented Jun 27, 2021 at 13:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.