I tried a similar technology stack, mines is based on Ubuntu
,Keycloak
, Prometheus
and spring boot
with a connection type of oauth-2.0. Its not the same, but some of the issues may be similar.
My first problem was that my version of Ubuntu had an ancient version of Prometheus
, which did not include the oauth support. Installing version 2.33, which has oauth-2.0 stuff included resolved this issue. The version of Prometheus
is returned by the '--version' flag.
To get Keycloak
to support a 'credit_credentials' its necessary to create a service_account, to support this, you must created a role of 'prometheus' and assigned it to the service_account
The Prometheus
config file prometheus.yml
is set up is as follows
- job_name: "keycloak"
metrics_path: '/actuator/prometheus'
scrape_interval: 5s
oauth2:
client_id: 'rm-config-server'
client_secret: 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
token_url: 'http://localhost:8085/auth/realms/BootAdmin/protocol/openid-connect/token'
endpoint_params:
grant_type: 'client_credentials'
static_configs:
- targets: ["localhost:8888"]
This says get a token from Keycloak
at address localhost:8085 with a realm of BootAdmin
and make a request for Prometheus
at the address localhost:8888. Where the XXXXX is a replacement for the actually value that Keycloak
generates for a client secret.
To see what Keycloak
generates, do a
curl --location --request POST 'http://localhost:8085/auth/realms/BootAdmin/protocol/openid-connect/token' --header 'Content-Type: application/x-www-form-urlencoded' --data-urlencode 'client_id=rm-config-server' --data-urlencode 'client_secret=XXXXXXXXXXXXXXXXX' --data-urlencode 'scope=email' --data-urlencode 'grant_type=client_credentials'
The result is token, if you paste the string following access_token into the debug box of JWT token decode it will decode what Keycloak
actually sent. In my case it sent a token with prometheus role defined.
The spring boot application includes in its KeycloakWebSecurityConfigurerAdapter method includes the line
requestMatchers(EndpointRequest.to("prometheus")).hasRole("PROMETHEUS")
Which says if you find a request for an endpoint ending in the string 'prometheus' with a role of 'ROLE_PROMETHEUS' accept it. My initial attempt here include the string '/acutator/prometheus' as the endpoint which resulted in an error.
By setting the logging level in spring boot to maximum and running Prometheus as prometheus --config.file prometheus.yml --log.level=debug
It was possible to see requests been made and 200 responses been returned.
Accessing the normal Prometheus
port of 9090, under pulls downs of Status/Service Discover
should show the name of the job if the connection has worked, in case it showed the job name of keycloak
In summary
Make sure your version of prometheus
includes the software support for oauth connections.
Make sure what ever token generating software your using is return what you expect it to. Do decode its output. In my case it several attempts to get the right information in to the generated token, something most client software will not display for security reasons.
Spy on the conversation that prometheus
is having with the debug option.
Remove all 'jobs' except the one that you are attempting to work on in the prometheus.yml.