6

Currently in my kubernetes-nodes job in Prometheus, The endpoint /api/v1/nodes/gk3-<cluster name>-default-pool-<something arbitrary>/proxy/metrics is being scraped

But the thing is I'm getting a 403 error which says GKEAutopilot authz: cluster scoped resource "nodes/proxy" is managed and access is denied when I try it manually on postman

How do I get around this on GKE Autopilot?

2
  • You need to use a valid ServiceAccount token with permission to access that subresource.
    – coderanger
    Commented May 19, 2021 at 3:59
  • @coderanger I already have a service account for prometheus that allows access to nodes/proxy with token intact on prometheus, this problem seems to be related to GKE's autopilot instead Commented May 19, 2021 at 5:01

4 Answers 4

5

While the Autopilot docs don't mention the node proxy API specifically, this is in the limitations section:

Most external monitoring tools require access that is restricted. Solutions from several Google Cloud partners are available for use on Autopilot, however not all are supported, and custom monitoring tools cannot be installed on Autopilot clusters.

Given that port-forward and all other node-level access is restricted it seems likely this is not available. It's not clear that Autopilot even uses Kubelet at all and they probably aren't going to tell you.

End of year update:

This mostly works now. Autopilot has added support for things like cluster-scope objects and webhooks. You do need to reconfigure any install manifests to not touch the kube-system namespace as that is still locked down but you can most of this working if you hammer on it a bunch.

1
  • Thanks, I think it makes sense that autopilot mode wants me to use GCP's own monitoring system since it's already automatically managed Commented May 19, 2021 at 9:11
2

Around June-July 2022 it seems this is now possible to do. However there's a lot of nuances. Most of the nuances are because if you read https://cloud.google.com/kubernetes-engine/docs/concepts/autopilot-overview#unsupported_cluster_features you'll find that GKE Autopilot clusters have a ton of restrictions in exchange for security, reliability, and a hands off UX(User Experience). There was also a change in GKE 1.22's default firewall rules, where unless a webhook uses port 443, you'll need to update firewall rules.


So I'll start with some background context:

There's at least 3 flavors of prometheus that can be deployed to GKE autopilot:

  1. Upstream Prometheus Operator that deploys a prometheus self hosted on the cluster.
  2. GCP's "managed collector" that sends metrics to GCP's Prometheus as a Managed Service
  3. GCP's "unmanaged collector" that sends metrics to GCP's Prometheus as a Managed service.

What is Upstream Prometheus Operator?
It's where Prometheus is a server with 2 roles: 1 it's a time series DB, 2 it scrapes / pulls metrics into itself to store in it's self hosted time series DB.

What is GCP's Prometheus as a Managed Service?
It's a Prometheus API compatibility layer on top of Monarch (GCP's in house time series DB they use for metrics / the same time series DB, the GCP platform's Metrics Explorer uses.)

What is GCP's collector?
It's a drop in replacement for prometheus server, but works different from how upstream prometheus server works. It's a thin wrapper that just collects prometheus metrics and then pushes them to GCP's managed prometheus service. (I suspect the difference between managed and unmanaged is just a kubernetes operator, and if you read the docs they recommend using the managed collector.)


Here's how you can get GCP's "managed collector" (their recommended) to deploy to a GKE Autopilot cluster, and verify they're going to GCP's managed prometheus service:

Provision a GKE Autopilot Cluster and then follow this page which says how to setup the managed collector https://cloud.google.com/stackdriver/docs/managed-prometheus/setup-managed#config-mgd-collection

It gives you 4 options to set it up: console, gcloud CLI, Terraform, and kubectl CLI. The first 3 options (console, gcloud CLI, and Terraform) won't work, trying will return an error saying it's unsupported. I suspect they all use the same API and I suspect the unsupported error message is out of date based on these 2 pages:
https://issuehint.com/issue/GoogleCloudPlatform/prometheus-engine/148
https://issuehint.com/issue/GoogleCloudPlatform/prometheus-engine/186
Ignore the warning and use the kubectl method and it'll work.

That page also has 3 commands you can run to generate some test data

kubectl create ns gmp-test

kubectl -n gmp-test apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/prometheus-engine/v0.4.3-gke.0/examples/example-app.yaml

kubectl -n gmp-test apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/prometheus-engine/v0.4.3-gke.0/examples/pod-monitoring.yaml

You can view the test data by doing the following:

kubectl get pod -o wide -n=gmp-test  

look up the IP address of a pod with metrics 10.8.0.132 in my case
then run

kubectl run curlpod -it --image=curlimages/curl -- sh
# got a message about timed out waiting for the condition which I ignored
kubectl exec -it curlpod -- curl 10.8.0.132:1234/metrics
# (I figured out :1234/metrics, by looking at the yaml manifest of the example pod)

The curl shows a big wall of text representing the prom metrics

example_random_numbers_bucket{le="0"} 0
...
go_memstats_stack_inuse_bytes 360448
...
process_virtual_memory_max_bytes -1

At this point things work correctly but if you go to Managed Service for Prometheus in the GCP's GUI Console, it's doesn't give immediate obvious feedback that things are working, due to the earlier step I know an example prom metric, so I plug go_memstats_stack_inuse_bytes into the GUI and say Run Query, and I'm able to verify it's working as expected.

1
Created a firewall to allow ingress traffic to port 10250-10255 (kubelet)
     $ gcloud compute firewall-rules create test-kubelet-ingress --allow tcp:10250-10255 --source-ranges="0.0.0.0/0"
Ran the following to:
### make sure the user can create nodes/proxy
  $  kubectl config view
  $ kubectl get all --all-namespaces
  $ kubectl create clusterrolebinding autopilot-cluster-1 --clusterrole=k8-cluster-1 [email protected]
### checking
   $ kubectl auth can-i create nodes/proxy
#> output
# Warning: resource 'nodes' is not namespace scoped
# yes
  $ curl -k https://{NODE_PUBLIC_IP}:10250/run/kube-system/{POD_NAME}/netd -d "cmd=ls" --header "Authorization: Bearer $TOKEN" --insecure
TOKEN = <auto generated token in local kubeconfig>
NODE_PUBLIC_IP = <the public ip of the node>
POD_NAME = <netd pod name in the node>
So even though the user has permissions in the kube-apiserver, it is denied to create a "nodes/proxy" by kubelet.
If nodes/proxy is removed from the authz, it success creating a proxy
$ curl -k https://35.202.254.215:10250/run/kube-system/netd-ff5vr/netd -d "cmd=ls" --header "Authorization: Bearer $TOKEN" --insecure
1

It looks like GKE Autopilot denies access to "nodes/proxy".

But it seems that Kubelet metrics are available. You can e.g. access them from within the cluster with:

curl  [Node_Internal_IP]:10255/metrics

I ended up with scraping the Kubelets directly, not via the proxy, using this scrape config:

- job_name: kubernetes-nodes
  kubernetes_sd_configs:
   - role: node
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  scheme: https
  metrics_path: /metrics/cadvisor

And you need this RBAC on a ClusterRole:

- apiGroups: [""]
  resources: ["nodes/metrics"]
  verbs: ["get"]

Using the above, it is possible to scrape container resource metrics from Kubelets in a GKE Autopilot cluster.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.