Around June-July 2022 it seems this is now possible to do. However there's a lot of nuances. Most of the nuances are because if you read https://cloud.google.com/kubernetes-engine/docs/concepts/autopilot-overview#unsupported_cluster_features you'll find that GKE Autopilot clusters have a ton of restrictions in exchange for security, reliability, and a hands off UX(User Experience). There was also a change in GKE 1.22's default firewall rules, where unless a webhook uses port 443, you'll need to update firewall rules.
So I'll start with some background context:
There's at least 3 flavors of prometheus that can be deployed to GKE autopilot:
- Upstream Prometheus Operator that deploys a prometheus self hosted on the cluster.
- GCP's "managed collector" that sends metrics to GCP's Prometheus as a Managed Service
- GCP's "unmanaged collector" that sends metrics to GCP's Prometheus as a Managed service.
What is Upstream Prometheus Operator?
It's where Prometheus is a server with 2 roles: 1 it's a time series DB, 2 it scrapes / pulls metrics into itself to store in it's self hosted time series DB.
What is GCP's Prometheus as a Managed Service?
It's a Prometheus API compatibility layer on top of Monarch (GCP's in house time series DB they use for metrics / the same time series DB, the GCP platform's Metrics Explorer uses.)
What is GCP's collector?
It's a drop in replacement for prometheus server, but works different from how upstream prometheus server works. It's a thin wrapper that just collects prometheus metrics and then pushes them to GCP's managed prometheus service. (I suspect the difference between managed and unmanaged is just a kubernetes operator, and if you read the docs they recommend using the managed collector.)
Here's how you can get GCP's "managed collector" (their recommended) to deploy to a GKE Autopilot cluster, and verify they're going to GCP's managed prometheus service:
Provision a GKE Autopilot Cluster and then follow this page
which says how to setup the managed collector
https://cloud.google.com/stackdriver/docs/managed-prometheus/setup-managed#config-mgd-collection
It gives you 4 options to set it up: console, gcloud CLI, Terraform, and kubectl CLI.
The first 3 options (console, gcloud CLI, and Terraform) won't work, trying will return an error saying it's unsupported. I suspect they all use the same API and I suspect the unsupported error message is out of date based on these 2 pages:
https://issuehint.com/issue/GoogleCloudPlatform/prometheus-engine/148
https://issuehint.com/issue/GoogleCloudPlatform/prometheus-engine/186
Ignore the warning and use the kubectl method and it'll work.
That page also has 3 commands you can run to generate some test data
kubectl create ns gmp-test
kubectl -n gmp-test apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/prometheus-engine/v0.4.3-gke.0/examples/example-app.yaml
kubectl -n gmp-test apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/prometheus-engine/v0.4.3-gke.0/examples/pod-monitoring.yaml
You can view the test data by doing the following:
kubectl get pod -o wide -n=gmp-test
look up the IP address of a pod with metrics 10.8.0.132 in my case
then run
kubectl run curlpod -it --image=curlimages/curl -- sh
# got a message about timed out waiting for the condition which I ignored
kubectl exec -it curlpod -- curl 10.8.0.132:1234/metrics
# (I figured out :1234/metrics, by looking at the yaml manifest of the example pod)
The curl shows a big wall of text representing the prom metrics
example_random_numbers_bucket{le="0"} 0
...
go_memstats_stack_inuse_bytes 360448
...
process_virtual_memory_max_bytes -1
At this point things work correctly but if you go to Managed Service for Prometheus in the GCP's GUI Console, it's doesn't give immediate obvious feedback that things are working, due to the earlier step I know an example prom metric, so I plug go_memstats_stack_inuse_bytes into the GUI and say Run Query, and I'm able to verify it's working as expected.
nodes/proxy
with token intact on prometheus, this problem seems to be related to GKE's autopilot instead