I'm considering exporting some metrics to Prometheus, and I'm getting nervous about what I'm planning to do.
My system consists of a workflow engine, and I'd like to track some metrics for each step in the workflow. This seems reasonable, with a gauge metric called wfengine_step_duration_seconds
. My issue is that there are many thousands of steps across all my workflows.
According to the documentation here, I'm not supposed to programmatically generate any part of the name. That precludes, then, the use of names such as wfengine_step1_duration_seconds
and wfengine_step2_duration_seconds
, because the step names are programmatic (they change from time to time).
The solution, then is a label for the step names. This also presents a problem, though, because the documentation here and here cautions quite strongly against using labels with high cardinality. Specifically, they recommend keeping "the cardinality of your metrics below 10", and for cardinality over 100, "investigate alternate solutions such as reducing the number of dimensions or moving the analysis away from monitoring".
I'm looking at a number of label values in the low thousands (1,000 to 10,000). Given that the number of metrics otherwise won't be extremely large, is this an appropriate usage of Prometheus, or should I limit myself to more generic metrics, such as a single aggregated step duration instead of individual duration for each step?