Questions tagged [prometheus]
The Prometheus monitoring system, including the server, alertmanager, push gateway, exporters, client libraries and other components.
5,254
questions
0
votes
0
answers
12
views
Grafana prometheus set $__interval to 1s
I now have my $__interval on 2s because when I try setting it to 1s it won't display the query anymore. Why is 2s the lowest I can go?
irate(rx_bytes{instance="localhost:2000"}[2s]) works ...
1
vote
0
answers
13
views
Creating Grafana alert based on string matching
I got a log result from a query in Grafana and I want to create an alert if a certain string is present in the logs.
For example: if a certain log contains the sentence "Task failed" then ...
0
votes
0
answers
6
views
Which Prometheus metrics type to use for reporting purchase amount
Which prometheus metrics type to use for reporting cart order / purchase amount? Should I use gauge or histogram?
0
votes
1
answer
20
views
How to configure istio helm chart to use external kube-prometheus-stack?
I have deployed the istio service mesh on the GKE cluster using base & istiod helm charts using this documents in the istio-system namespace.
I have deployed Prometheus, grafana & alert-...
0
votes
1
answer
9
views
why prometheus using the rate to calculate the request_duration
since the request_duration is just a counter, why we need to using the rate to calculate the duration, this is not meaningful.
histogram_quantile(0.99, sum by (le) (rate(...
0
votes
1
answer
8
views
Which type in Prometheus is best for DAU representation?
DAU is always incremental throughout the day, so it looks like Counter is better and easier to calculate increments.
However, DAU is cleared again every day, so it does not satisfy the requirement of ...
0
votes
0
answers
14
views
csi-cephfs: how can I get ceph storage cluster available capacity from kubernetes
My goal is to get the capacity of ceph cluster by ceph-csi in kubernetes
Same as the command in ceph 'ceph -s'
usage: 10 Gib used, 12GiB /22 GiB avail
Version is
ceph-csi v3.6.0
kubernetes v1.21....
0
votes
1
answer
19
views
Importing Grafana dashboard - label_values doesn't work with InfluxDb
I found nice Grafana dashboard for Resilience4j metrics: https://resilience4j.readme.io/docs/grafana-1
The problem is that I am using InfluxDb data source but the dashboard was created for Prometheus.
...
0
votes
1
answer
21
views
How to export/get the Alerting Rules from prometheus using Python
We have more than 200 alerts created on prometheus. I want to somehow get/export them one by one to a txt file. I want to automate that.
Can you recommend me any way of doing that?
0
votes
0
answers
9
views
Prometheous scrape is not working on custom service monitor
I'm trying to add one service monitor for the one the service using the selector Selector: app=solana-exporter in kubernetes.
Kuberenetes is deployed with helm chart kube-prometheus-stack
...
-1
votes
0
answers
13
views
is it possible to monitor containers From another Server Using Prometheus?
Not sure, I am really new with Prometheus.
I just have a case:
I have many servers just call it three servers (A,B, and C). all servers running with Ubuntu.
Each server has many running docker ...
0
votes
0
answers
38
views
Spring Actuator Prometheus: visualize available metrics
I have an endpoint /api/actuator/prometheus that returns registered metrics in Prometheus format.
Is it possible to format this document prettily and host it on GitLab Pages?
I want to do something ...
0
votes
0
answers
13
views
What is the correct way to add custom Prometheus metrics to a client library?
We have recently started monitoring our springboot apps with the micrometer prometheus integration. For the most part it works great out of the box; however, we have a client lib which wraps an http ...
-1
votes
0
answers
8
views
Is there an equivalent of "Ping_exporter" of prometheus to monitor windows targets
I want to monitor the ping status of a windows system. Is there a windows specific version of ping_exporter ?
Or is there another better way to monitor windows target ?
0
votes
0
answers
13
views
Data loss after promethues restart
whenever we are restarting the Prometheus service it causes data loss of 2-3 hours.
This was not happening previously, but when we started adding more data/integration into it, we are experiencing ...
0
votes
0
answers
10
views
PromQL expression to find common labels in two buckets
I have metrics collected every 30m, so for example in a 60m interval i have two groups each 30m interval -
group 1
fruits{name="apple"} 1
fruits{name="orange"} 1
fruits{name="...
0
votes
0
answers
10
views
Get Prometheus metrics from flask which is running in kubernates pods
I have integrated Prometheus with my flask application and my application is running in kubernates pods(2 pods).
I have attached same volume for both pods. When i hit "/metrics" endpoint i'm ...
0
votes
0
answers
20
views
Define grafana variable as prometheus query
I am planning to add a variable in grafana template using a prometheus query so I can use this variable to plot the graph in a grafana dashboard.
I am using this Query
query_result((vector(1) and on() ...
0
votes
0
answers
20
views
Caching Tokens in Prometheus
I am using OAuth 2.0 with Prometheus, and don't want to generate a new access token every single scrape (which is too often) - I want to generate it once every hour. How can I use a caching system in ...
0
votes
1
answer
23
views
Encrypting a Secret String in a Prometheus YAML file
I am using OAuth2.0 in this Prometheus YAML file, and don't want to expose the client_secret directly in the file. Does anybody know how to encrypt another file with the client secret (...
0
votes
1
answer
9
views
PromQL delta over time
We would like to see the difference of exposed metrics by a combination of node/ip over time. Nodes or ips could exist at any given point but not neccesarily will exist in another.
The metric itself (...
0
votes
0
answers
23
views
Spring Kafka monitoring in Grafana
As showcased in the following https://docs.spring.io/spring-kafka/docs/current/reference/html/#micrometer and https://docs.spring.io/spring-kafka/docs/current/reference/html/#streams-micrometer, ...
0
votes
0
answers
18
views
How is the name field of container_cpu_usage_seconds_total derived?
I am using the dashboard in https://grafana.com/grafana/dashboards/4784. Metrics like container_cpu_usage_seconds_total is filtered with name=~"^k8s_.*" but not all the metrics have this ...
0
votes
1
answer
16
views
Trying to turn a time series into stat on Grafana
I'm new to Grafana and I'm trying to turn a time series that I have into a Stat on Grafana, but when I make it, I got zero.
The below picture shows how is the graph when I select the time series.
...
0
votes
0
answers
27
views
container's runAsUser breaks non-root policy in prometheus helm chart (wireguard sidecar)
I want to add a wireguard sidecar container to prometheus stack installed via helm chart such that I can fetch client connected via vpn. I'm using the helm chart from prometheus-community/kube-...
0
votes
0
answers
11
views
Promquery to calculate pod uptime percentage - (Without Up Metric)
I am trying to create a visualization in Grafana that would display the pod uptime percentage over a period of time (30 days) for all the pods in the namespace. For some reason, We are unable to use ...
0
votes
0
answers
16
views
Is it possible to aggregate based on some label, but include all the label in the prometheus alert(use case below)?
let's say I have few data points for the following metric
my_metric(error_message='error1', status='500', request_id='1234') 1
my_metric(error_message='error1', status='500', request_id='2234') 1
...
0
votes
0
answers
21
views
Prometheus Federated server metrics not appearing
I have multiple Kubernetes clusters and I wish to use the Prometheus federation.
Each cluster has its own Prometheus installed using helm.
Only one cluster's Prometheus is connected to Grafana.
This ...
0
votes
0
answers
11
views
Method for pushing/replicating/mirror Prometheus metrics between several network layers
Assume a dataflow as in the picture. Data is being transported from a location to the cloud via a data gateway and bridged MQTT brokers to do Stuff™. The firewalls allows you to initialize ...
0
votes
0
answers
10
views
Decreasing `Min interval` for Grafana hides the data completely [duplicate]
I have a standard Grafana installation bundled with a standard Prometheus configuration. Also, I have my own exporter (which I plan to make open-source) for the SendPulse service, which requests the ...
0
votes
1
answer
22
views
How to get percentage of time that CPU usage is higher than x%?
I have this as a query to get CPU usage in percentage:
avg without (mode,cpu) (
1 - rate(node_cpu_seconds_total{mode="idle", job="openwrt"}[1m])
) *100
I want to get the ...
0
votes
0
answers
10
views
prometheus `kube_node_status_condition merged * kube_node_status_condition` returns many nodes
Using the following prometheus images inside EKS 1.22:
alertmanager:v0.23.0
kube-state-metrics:v2.4.1
prometheus/node-exporter:v1.3.0
prometheus:v2.34.0
I'm trying to get labels of the node that is ...
0
votes
1
answer
32
views
"Could not find expected ':' " error on Prometheus YAML file
I am trying to configure a YAML using oauth2.0 and I keep getting a "Could not find expected ':'" error on the line towards the end that says "scopes."
# my global config
global:
...
1
vote
1
answer
44
views
kube-state-metrics not sending metrics using service monitor
I have deployed kube-state-metrics into kube-system namespace and in the same cluster we are having prometheus-operator running I've written the below service monitor file for sending metrics to ...
1
vote
1
answer
28
views
Grafana: duplicate Legend and Graph for Rate Query after Pod Restart
I want to show the rate of logins per hour in a web app. This is the PromQL query I have:
rate(logins_total[1h])*3600
The situation in the database is that there has been a restart of the pod running ...
0
votes
0
answers
21
views
Count Number of Request in Grafana
My Project is pushing these metrics.
About the metrics :
``zs_http_response is a Histogram for HTTP response times in seconds
Label Description
path. path for the HTTP Request
method ...
0
votes
0
answers
27
views
Podmetrics not getting scraped into prometheus-operator
I am currently working on setting up PodMetrics on a sample application Link
The application is deployed fine this is the deployment.yaml file I am using currently
apiVersion: apps/v1
kind: Deployment
...
0
votes
0
answers
17
views
Prometheus scraping only some metrics
In my cluster I have an nginx ingress, nodejs server and prometheus. Prometheus is scraping all metrics from nginx, no problem, but it seems that it's ommitting some metrics from my nodejs server.
# ...
0
votes
0
answers
27
views
How to configure datasource of Grafana using bitnami helm charts?
I am having trouble with configuring grafana datasource with Helm Charts and Kubernetees. This is my release.yaml:
kind: Namespace
apiVersion: v1
metadata:
name: monitoring
annotations:
name: ...
0
votes
0
answers
14
views
Can ServiceMonitor add labels to the metric?
I have a requirement to add certain labels to metric, so grafana can filter based on its value. I am wondering if there is a way to do it in the ServiceMonitor, so that I dont need to change the code ...
0
votes
0
answers
23
views
Prometheus Execution Timeout Exceeded
I use Grafana to monitor my company's infrastructure. Everything worked fine until this week, I started to see alerts on Grafana with an error message :
request handler error: Post "http://...
0
votes
0
answers
9
views
Prometheus promtool unittest how to pass series label as range?
i have a below alert expr:
- alert: Switch_link_between_a_to_b_down
expr: (switch_interface_st{name=~"et-0/1/3",status="UP"} offset 10m) and (switch_interface_st{name=~&...
0
votes
0
answers
16
views
How to get current CPU usage value in prometheus?
Could you help me set up a query to get the current CPU value in Prometheus?
Thank you.
0
votes
0
answers
4
views
Prometheus: customize generic webhook_configs payloads
Need
In order to avoid integrating third party operators, I am looking for a way to template messages send by Prometheus' alertmanager.
Here is the webhook_config documentation. It suggests that ...
0
votes
0
answers
11
views
Prometheus prom file is not seen my metrics
Hello I have 5 computers with windows exporter .
The metrics are working on all of them and I have a script which is generating a .prom file which is working only on 3 on them.
I really cant figure ...
0
votes
0
answers
15
views
How to create a pivot graph in grafana using a prometheus query
I would like to know if is possible join/merge 2 metric query from Prometheus data source for a rendering graph in grafna like a PIVOT, grouped by a LABELA with 2 value, like:
D 1500 1000
E 4444 112
T ...
0
votes
1
answer
42
views
How to redirect Prometheus Metrics to the default spring boot server
I am trying to expose a custom Gauge metric from my Spring Boot Application. I am using Micrometer with the Prometheus registry to do so. I have set up the PrometheusRegistry and configs as per - ...
0
votes
0
answers
24
views
Prometheus Remote Read Error: remote_read: remote server (url) returned HTTP status 405 Method Not Allowed: Method Not Allowed
I've set up a Prometheus RemoteReadSpec in an unmanaged prometheus namespace. I've set the following fields:
url as the internal url for thanos' query frontend
name as thanos
requiredMatchers as ...
0
votes
1
answer
38
views
How to add the scrape_configs file to Prometheus
I've deployed Prometheus, Alertmanager, Grafana et cetera in the cattle-monitoring-system namespace using the rancher-monitoring Helm chart. Look here and here for documentation. Everything in this ...
1
vote
2
answers
47
views
Azure Kubernetes Service Scaling and HPA
Is there a way that "request per second" based scaling can be enabled on Azure Kubernetes Service?
HPA in AKS does allow horizontal pod scaling based on CPU and Memory constraints but there ...