I have container_fs_usage_bytes
with prometheus to monitor container root fs, but it seems that there is no metrics for other volumes in cAdvisor.
-
1There is a ticket in the community to work on the PVC disk usage data, here is the link, fyi: github.com/kubernetes/features/issues/293– Tom XingJun 28, 2017 at 7:59
3 Answers
I confirmed that Kubernetes 1.8 expose metrics for prometheus.
kubelet_volume_stats_available_bytes
kubelet_volume_stats_capacity_bytes
kubelet_volume_stats_inodes
kubelet_volume_stats_inodes_free
kubelet_volume_stats_inodes_used
kubelet_volume_stats_used_bytes
-
1If you use Grafana, this dashboard shows the information graphically: grafana.com/dashboards/6739– RamanOct 28, 2018 at 19:38
-
22
-
5
-
These metrics only only contain the information for PVCs created using the default storageclass. How can I view information for the PVCs created using other storageclasses?– HemabhJul 13, 2022 at 6:49
Metrics for volumes are available via the kubelet summary API (/stats/summary). However, each volume plugin has to implement their own metrics. As of Kubernetes 1.7, the current volume plugins that have implemented metrics include: emptydir, secrets, gce pd, aws ebs, azure file, flocker, and portworx
-
I don't understand what this means yet. I'll find out /stats/summary API. I'm using GKE and gce pd. Will it be available through cAdvisor and prometheus after they rolling out kubernetes 1.7 to GKE?– hiroshiJun 29, 2017 at 0:23
The following metrics must be used for monitoring persistent volume stats in Kubernetes (the PVC name is exported in persistentvolumeclaim
label):
kubelet_volume_stats_capacity_bytes
- the per-PVC capacity in bytes.kubelet_volume_stats_used_bytes
- the per-PVC space usage in bytes.kubelet_volume_stats_available_bytes
- the per-PVC free space in bytes.
The following PromQL queries can be used for determining per-pod PVC disk space usage in bytes:
sum(kubelet_volume_stats_used_bytes) by (namespace,persistentvolumeclaim)
* on(namespace,persistentvolumeclaim) group_left(pod)
kube_pod_spec_volumes_persistentvolumeclaims_info
The following query can be used for determining free PVC disk space in bytes per each pod:
sum(kubelet_volume_stats_available_bytes) by (namespace,persistentvolumeclaim)
* on(namespace,persistentvolumeclaim) group_left(pod)
kube_pod_spec_volumes_persistentvolumeclaims_info
-
In my case,
kubelet_volume_stats_used_bytes
only contains the metrics for PVCs created with the default storageclass (gp2). I have other PVCs attached to an efs storageclass as well. How can I view their metrics?– HemabhJul 13, 2022 at 6:48 -
-
1Thank you so much! I just to mention that
kubelet_
metrics are exported by kubelet ^^ andkube_pod_
metrics are exported by kube-state-metrics. If using kube-prometheus-stack helm chart, both components can be easily enabled. Nov 6, 2022 at 9:32