18

I have container_fs_usage_bytes with prometheus to monitor container root fs, but it seems that there is no metrics for other volumes in cAdvisor.

1

3 Answers 3

38

I confirmed that Kubernetes 1.8 expose metrics for prometheus.

  • kubelet_volume_stats_available_bytes
  • kubelet_volume_stats_capacity_bytes
  • kubelet_volume_stats_inodes
  • kubelet_volume_stats_inodes_free
  • kubelet_volume_stats_inodes_used
  • kubelet_volume_stats_used_bytes
4
  • 1
    If you use Grafana, this dashboard shows the information graphically: grafana.com/dashboards/6739
    – Raman
    Oct 28, 2018 at 19:38
  • 22
    They no longer seem to be exposed starting from 1.12 Feb 13, 2019 at 22:23
  • 5
    @AndrewSavinykh Are there any new Volume usage metrics?
    – Amrit
    Oct 10, 2019 at 11:10
  • These metrics only only contain the information for PVCs created using the default storageclass. How can I view information for the PVCs created using other storageclasses?
    – Hemabh
    Jul 13, 2022 at 6:49
7

Metrics for volumes are available via the kubelet summary API (/stats/summary). However, each volume plugin has to implement their own metrics. As of Kubernetes 1.7, the current volume plugins that have implemented metrics include: emptydir, secrets, gce pd, aws ebs, azure file, flocker, and portworx

1
  • I don't understand what this means yet. I'll find out /stats/summary API. I'm using GKE and gce pd. Will it be available through cAdvisor and prometheus after they rolling out kubernetes 1.7 to GKE?
    – hiroshi
    Jun 29, 2017 at 0:23
4

The following metrics must be used for monitoring persistent volume stats in Kubernetes (the PVC name is exported in persistentvolumeclaim label):

  • kubelet_volume_stats_capacity_bytes - the per-PVC capacity in bytes.
  • kubelet_volume_stats_used_bytes - the per-PVC space usage in bytes.
  • kubelet_volume_stats_available_bytes - the per-PVC free space in bytes.

The following PromQL queries can be used for determining per-pod PVC disk space usage in bytes:

sum(kubelet_volume_stats_used_bytes) by (namespace,persistentvolumeclaim)
  * on(namespace,persistentvolumeclaim) group_left(pod)
kube_pod_spec_volumes_persistentvolumeclaims_info

The following query can be used for determining free PVC disk space in bytes per each pod:

sum(kubelet_volume_stats_available_bytes) by (namespace,persistentvolumeclaim)
  * on(namespace,persistentvolumeclaim) group_left(pod)
kube_pod_spec_volumes_persistentvolumeclaims_info
3
  • In my case, kubelet_volume_stats_used_bytes only contains the metrics for PVCs created with the default storageclass (gp2). I have other PVCs attached to an efs storageclass as well. How can I view their metrics?
    – Hemabh
    Jul 13, 2022 at 6:48
  • EFS has no upper size limits, e.g. the kubelet_volume_stats_available_bytes metric cannot be defined for EFS. Probably, that's why other kubelet_volume_stats_* metrics aren't exported for EFS volumes.
    – valyala
    Jul 13, 2022 at 15:57
  • 1
    Thank you so much! I just to mention that kubelet_ metrics are exported by kubelet ^^ and kube_pod_ metrics are exported by kube-state-metrics. If using kube-prometheus-stack helm chart, both components can be easily enabled.
    – Nour Wolf
    Nov 6, 2022 at 9:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.