5

I'm using ingress-nginx as an Ingress controller for one of my services running over K8S (I'm using the nginx-0.20.0 release image with no specific metrics configurations in the K8S configmap the ingress-controller is using).

The nginx-ingress-controller pods are successfully scraped into my Prometheus server but all ingress metrics (e.g. nginx_ingress_controller_request_duration_seconds_bucket) show up with path="/" regardless of the real path of the handled request.

Worth noting that when I look at the ingress logs - the path is logged correctly.

How can I get the real path noted in the exported metrics?

Thanks!

2
  • Have you enabled vts metrics: enable-vts-status: "true"?
    – Nick_Kh
    Commented Feb 11, 2019 at 15:09
  • @mk_sta I tried it and it didn't seem to have any effect :( but thanks!
    – wilfo
    Commented Feb 14, 2019 at 11:19

2 Answers 2

5

The Path attribute in the NGINX metrics collected by prometheus derives from the Ingress definition yaml.

For example, if your ingress is:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
  name: <some-k8s-ingress-name>
  namespace: <some-k8s-namespace-name>
spec:
  rules:
  - host: <hostname>
    http:
      paths:
      - backend:
          serviceName: <some-k8s-service-name>
          servicePort: <some-port>
        path: /

Then although NGINX will match any URL to your service, it'll all be logged under the path "/" (as seen here).

If you want metrics for a specific URL, you'll need to explicitly specify it like this (notice the ordering of rules):

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
 annotations:
   kubernetes.io/ingress.class: nginx
 name: <some-k8s-ingress-name>
 namespace: <some-k8s-namespace-name>
spec:
 rules:
 - host: <hostname>
   http:
     paths:
     - backend:
         serviceName: <some-k8s-service-name>
         servicePort: <some-port>
       path: /more/specific/path
     - backend:
         serviceName: <some-k8s-service-name>
         servicePort: <some-port>
       path: /
1
  • and thanks to evgenyl for leading me to understanding this!
    – wilfo
    Commented Feb 19, 2019 at 13:37
1

If I am understand right, this label was removed as causing high cardinality: https://github.com/kubernetes/ingress-nginx/issues/2924#issuecomment-424516095

And this is more general issue: https://github.com/kubernetes/ingress-nginx/pull/2701

Have to say that I am pretty agree - think about it: if you allow the route to be the label value, it can be pretty easy to DDoS your Prometheus - just call your service with a lot of unexisting routes.

All those calls will be translated into the Label Values for all ingress metrics....

1
  • Thanks for the comment! I've gone through the threads and discovered the ingress controller behaves differently than I thought. In my ingress definition I used path: / to catch all calls to the service while thinking the controller would capture the exact path. But, it doesn't (not even with regex, from my trials). If I want the metrics to register for a specific path, I need to explicitly add a rule for that path. I'll post this as an answer.
    – wilfo
    Commented Feb 19, 2019 at 13:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.