0

Recently added to prometheus the possibility to scrape targets using oauth2 authentication. And when using it I get the following error, it looks like the Authorization is not being sent.

    Get "http://target.foo.bar.net:80/my-service/metrics": oauth2: cannot fetch token: 500 Internal Server Error Response: {"fault":{"faultstring":"Unresolved variable : request.header.Authorization","detail":{"errorcode":"steps.basicauthentication.UnresolvedVariable"}}}

This is my configuration from prometheus.yml.

Where token_url is the address of my service that authenticates my client_id and client_secret and responds with a token. And targets my endpoint where I will monitor authenticating using the token.

  scrape_configs:
    - job_name: my-service
      metrics_path: /target.foo.bar/metrics
      oauth2:
        client_id: 'CLIENT_ID'
        client_secret: 'CLIENT_SECRET'
        token_url: 'https://token-foo-generator.com/oauth/v1/token'
        endpoint_params:
          grant_type: 'client_credentials'
      static_configs:
        - targets:
            - target.foo.bar.net

As this was recently added to prometheus(like two months ago), I found little documentation on this and I don't know if I'm doing it correctly.

1 Answer 1

0

I tried a similar technology stack, mines is based on Ubuntu,Keycloak, Prometheus and spring boot with a connection type of oauth-2.0. Its not the same, but some of the issues may be similar.

My first problem was that my version of Ubuntu had an ancient version of Prometheus, which did not include the oauth support. Installing version 2.33, which has oauth-2.0 stuff included resolved this issue. The version of Prometheus is returned by the '--version' flag. To get Keycloak to support a 'credit_credentials' its necessary to create a service_account, to support this, you must created a role of 'prometheus' and assigned it to the service_account

The Prometheus config file prometheus.yml is set up is as follows

- job_name: "keycloak"
metrics_path: '/actuator/prometheus'
scrape_interval: 5s
oauth2:
  client_id: 'rm-config-server'
  client_secret: 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
  token_url: 'http://localhost:8085/auth/realms/BootAdmin/protocol/openid-connect/token'
  endpoint_params:
    grant_type: 'client_credentials'
static_configs:
  - targets: ["localhost:8888"]

This says get a token from Keycloak at address localhost:8085 with a realm of BootAdmin and make a request for Prometheus at the address localhost:8888. Where the XXXXX is a replacement for the actually value that Keycloak generates for a client secret.

To see what Keycloak generates, do a

curl --location --request POST 'http://localhost:8085/auth/realms/BootAdmin/protocol/openid-connect/token' --header 'Content-Type: application/x-www-form-urlencoded' --data-urlencode 'client_id=rm-config-server' --data-urlencode 'client_secret=XXXXXXXXXXXXXXXXX' --data-urlencode 'scope=email' --data-urlencode 'grant_type=client_credentials'

The result is token, if you paste the string following access_token into the debug box of JWT token decode it will decode what Keycloak actually sent. In my case it sent a token with prometheus role defined.

The spring boot application includes in its KeycloakWebSecurityConfigurerAdapter method includes the line

requestMatchers(EndpointRequest.to("prometheus")).hasRole("PROMETHEUS")

Which says if you find a request for an endpoint ending in the string 'prometheus' with a role of 'ROLE_PROMETHEUS' accept it. My initial attempt here include the string '/acutator/prometheus' as the endpoint which resulted in an error.

By setting the logging level in spring boot to maximum and running Prometheus as prometheus --config.file prometheus.yml --log.level=debug

It was possible to see requests been made and 200 responses been returned.

Accessing the normal Prometheus port of 9090, under pulls downs of Status/Service Discover should show the name of the job if the connection has worked, in case it showed the job name of keycloak

In summary

  1. Make sure your version of prometheus includes the software support for oauth connections.

  2. Make sure what ever token generating software your using is return what you expect it to. Do decode its output. In my case it several attempts to get the right information in to the generated token, something most client software will not display for security reasons.

  3. Spy on the conversation that prometheus is having with the debug option.

  4. Remove all 'jobs' except the one that you are attempting to work on in the prometheus.yml.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.