I've an application, and I'm running one instance of this application per AWS region.
I'm trying to instrument the application code with Prometheus metrics client, and will be exposing the collected metrics to the /metrics
endpoint. There is a central server which will scrape the /metrics
endpoints across all the regions and will store them in a central Time Series Database.
Let's say I've defined a metric named: http_responses_total
then I would like to know its value aggregated over all the regions along with individual regional values.
How do I store this region
information which could be any one of the 13 regions and env
information which could be dev
or test
or prod
along with metrics so that I can slice and dice metrics based on region
and env
?
I found a few ways to do it, but not sure how it's done in general, as it seems a pretty common scenario:
- Storing
region
andenv
info as labels with each of the metrics (not recommended: https://prometheus.io/docs/instrumenting/writing_exporters/#target-labels-not-static-scraped-labels) - Using target labels - I have
region
andenv
value with me in the application and would like to set this information from the application itself instead of setting them in scrape config - Keeping a separate gauge metric to record
region
andenv
info as labels (like described here: https://www.robustperception.io/exposing-the-software-version-to-prometheus) - this is how I'm planning to store my applicationversion
info in tsdb but the difference between appversion
info andregion
info is: the version keeps changing across releases however region is which I get from the config file is constant. So, not sure if this is a good way to do it.
I'm new to Prometheus. Could someone please suggest how I should store this region
and env
information? Are there any other better ways?