7

Right now I'm scraping metrics from a Node Exporter. Some of the metrics which the Node Exporters exports have a mountpoint label.

I'd like to drop time series that have this label and do not match a regular expression. I tried using the keep action (as I'd like to keep time series that do match this regular expression) but this also drops all other metrics that do not have the mountpoint label.

metric_relabel_configs:
  - source_labels: ['mountpoint']
    regex: '(\/home|\/var\/domains)\/something.*'
    action: keep

I tried using the drop action too but this requires me to inverse the regular expression using a negative-lookahead (which isn't supported because Prometheus is written in Go of course).

What are my options in this?

Important, I do not have control over the way the Node Exporter is configured, thus I can't configure the Node Exporter itself to not export metrics for some specific mountpoint (if this is even possible).

2 Answers 2

3

Just add | to the end of the regex, so it matches time series without mountpoint label:

metric_relabel_configs:
- source_labels: [mountpoint]
  regex: '(/home|/var/domains)/something.*|'
  action: keep

Shameless plug (I'm the author of vmagent): vmagent supports easier-to-understand-and-use construct for this particular case with if option:

metric_relabel_configs:
- if: '{mountpoint=~"/home|/var/domains)/something.*|"}'
  action: keep

The if option may contain arbitrary time series selector. See these docs for more details.

2

If I understand you correctly, then the following should meet your needs. I tested it in the Relabeler online tool and seems to do what you want?

Using these relabeling rules:

- source_labels: ['mountpoint']
  regex: '(\/home|\/var\/domains)\/something.*'
  target_label: __tmp_keep_me
  replacement: true
- source_labels: [__tmp_keep_me]
  regex: true
  action: keep

The following (example) object labels would be kept:

mountpoint: "/home/something/"
job: "node"
fstype: "ext4"

While these would be dropped:

mountpoint: "/tmp"
job: "node"
fstype: "ext4"

Note that I used a tip Brian Brazil shared in the Or in relabelling article and adapted it.

3
  • 2
    It does not because the problem still exists that time series without the "mountpoint" label would get dropped as well. I didn't know about the "_tmp" convention / relabeling technique so thanks for that!
    – Julian
    Commented Jun 18, 2021 at 13:15
  • Roger that. Maybe you wanna ask this in discuss.prometheus.io … I'm sure someone will have an answer there ;) Commented Jun 18, 2021 at 13:48
  • 1
    Will do that, thanks so much for the help! Really appreciate it a lot ;-)
    – Julian
    Commented Jun 18, 2021 at 14:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.