Questions tagged [observability]
monitoring the internal state of a system by looking at its output
91
questions
2
votes
0
answers
197
views
Exception in observability stack after upgrading Spring Boot from 3.1.3 to 3.2.0 using opentelemetry-java
Problem:
When upgrading the observability stack from Spring Boot version 3.1.3 to 3.2.0 using OTel Java SDK, the following exception is encountered:
2023-11-26T16:00:43.087Z WARN [order service,,] 1 -...
0
votes
0
answers
14
views
Observing subprocess execution times and memory usages, should it be a Metric? or an Event?
I have an application that runs various subprocesses dynamically during its runtime. I aim to create graphs depicting the execution time of these subprocesses and their corresponding memory usage. ...
0
votes
1
answer
97
views
Collecting data with Dynatrace OneAgent or OpenTelemetry
I am currently thinking about debating using OpenTelemetry to collect data and then send it to Dynatrace or collecting data with the OneAgent.
Have you ever made experiences with this?
My question is, ...
-1
votes
1
answer
35
views
Using ELK in Azure Cloud
We are using Azure App Insights for all tracing . monitoring and log requirements of our applications running in Azure.
We have seen cases where teams ship the logs to ELK for observability. We do not ...
1
vote
0
answers
41
views
How to get request/response body with uprobe?
https://github.com/open-telemetry/opentelemetry-go-instrumentation
internal/pkg/instrumentation/bpf/net/http/server/probe.go
It injects a lot of fields,but Event only contains part of. How can I get ...
0
votes
1
answer
43
views
I got an alert for a latency spike in my Cloud Spanner database. How can I debug it?
I am using Cloud Spanner as the database for my application. For certain queries the latency has spiked. I would like to know how can I debug it?
I tried looking at the system insights page but couldn'...
0
votes
1
answer
99
views
fluentd vs prometheus vs elasticsearch, when to use what?
When to use fluentd vs prometheus vs elasticsearch?
There seems to be overlapping between these tools interms of the feature set they provide but I have seen projects using all these 3 tools in unison....
0
votes
1
answer
30
views
How to make a Ballerina function observable?
In Ballerina, the main, resource, and remote functions come inherently equipped with observability to enable metrics and tracing monitoring. However, I wish to extend this observability to some of the ...
2
votes
1
answer
70
views
Parse string of key=values with escaped characters
Loki outputs the following log in a key-value format with the structure key1=value1 key2=value2:
level=info ts=2023-10-20T14:30:48.716410806Z caller=metrics.go:159 component=frontend org_id=fake ...
0
votes
1
answer
52
views
Spring Observability questions
I implemented code from this article:
https://programmingtechie.com/2023/09/09/spring-boot3-observability-grafana-stack/
Can I enable tracing only for specific endpoints ?
Trace id between services ...
1
vote
1
answer
50
views
OpenTelemetry adjust sampling ratio programmatically
In the OpenTelemetry SDK for JS, is there a way to modify the TraceIdRatioBasedSampler sampling ratio at run time? Alternatively, I was considering creating a new sampler and updating the trace ...
2
votes
0
answers
71
views
Does wekbit have an equivalent performance API to chrome for measuring memory usage?
Chrome has measureUserAgentSpecificMemory, which "is used to estimate the memory usage of a web application including all its iframes and workers."
Is there an equivalent for Safari?
The ...
0
votes
0
answers
74
views
How to write query in PromQL and assign it to a variable for alert conditions?
I have a below query from SignalFx and i need to convert it to PromQL:
A = data('signal_apiserver_graphql.errors_count', filter=filter('kubernetes_cluster', 'dev-001', 'dev-003', 'dev-002', 'dev-000'...
0
votes
0
answers
25
views
How to integrate AWS X-ray with this format StepFunction => EventBridge => Downstream services
How to include EventBridge Triggered Downstream Services in X-ray Service Map?
I have a step function with tracing enabled, and within this step function, there's an EventBridge integration that both ...
0
votes
0
answers
20
views
Unable to export spans from OTEL to Elastic APM Server
I have a nestjs application that i want to instrument. I am collecting traces using OTEL and exporting it to zipking and elastic apm server. My main focus is to trace all my graphql queries. I am able ...
0
votes
0
answers
7
views
How can I deliver custom micrometer configuration for use by corporate internal microservices?
I want our internal corporate microservices to use recommended Observability configuration merely by adding in the build dependency.
One way to do this is with Spring Boot starter. Assume the teams ...
0
votes
0
answers
25
views
Observability for DLQs? [AWS SQS]
I'm trying to understand what is the best way to implement some sort of Observability for DLQs.
I know it's possible to have some monitoring (e.g.: There are 3 messages in the DLQ XYZ)
I'm trying to ...
0
votes
1
answer
33
views
unknown field `healthcheck` with Vector
I am trying to expose the endpoint "http://localhost:8686/health" using Vector so that I can monitor the Vector's health.
After research, I came across guides that recommend putting the ...
0
votes
1
answer
138
views
Unable to send open telemetry traces to OpenSearch
I am trying to send open telemetry traces to OpenSearch but I am unable to. The traces are visible on Jeager UI but they are not getting ingested in OpenSearch .I have tried everything ,created ...
0
votes
0
answers
39
views
How to export logs for certain interval from Kibana Observability/Logs/Stream?
We need to export logs from the stream for a specific time interval (which is set on the top of the main stream frame) into JSON, CSV or any other human-readable format.
The logs "live" 7 ...
0
votes
1
answer
37
views
DataDog alert includes filtered out environment tag
I have a DataDog query alert like so:
min(last_5m):sum:my_service.verification.failure{service:"my_service" AND env:"production"} by {region,env} >= 1
And I enabled the ...
0
votes
0
answers
55
views
Why use Open Telemetry Logging in Kubernetes dotnet service?
I have a dotnet web api that has been deployed to a Kubernetes cluster.
I see a lot of experienced folks suggesting using Open Telemetry SDK with Console exporter instead of dotnet inbuilt logging ...
0
votes
0
answers
17
views
Spring boot Observablity for Event lifecycle
I am creating a application with spring boot 3 and its observability feature using micrometer and actuators.
What my application does is, it receives a alert, generates event based on this alert, ...
0
votes
1
answer
686
views
Axios HTTP backend call generates Opentelemetry span that is not correlated with the parent span
I'm trying to perform manual instrumentation in NodeJs (v18.16.1) using opentelemetry. The scenario is very simple, I'm making an axios HTTP call to a dummy JSON API and I want to collect the ...
1
vote
1
answer
1k
views
Error Loki distributed with Helm: no private IP address found, and explicit IP not provided
I'm configuring loki-distributed on a kubernetes cluster via helm, but I'm getting the following error:
failed to create memberlist: Failed to get final advertise address: no private IP address found, ...
1
vote
1
answer
388
views
Dependencies do not show with OpenTelemetry automatic instrumentation for PHP
I have a very simple stack of PHP + Otel auto instrumetation + Elastic APM + Prometheus.
My stack gets tracing and metrics information from my php app.
My tracings are forwarded to the elastic cloud, ...
0
votes
0
answers
26
views
Need to integrate Splunk with Git such that Splunk can clone and commit to Git
If Splunk detects certain events, it needs to trigger cloning, committing, and creating a pull request on Git.
I can find add-ons on Splunkbase that let me send data from Git to Splunk, but is there ...
0
votes
1
answer
141
views
How do i set response_header_timeout in Thanos Query Frontend Manifest?
I want to increase the response_header_timeout in thanos query-frontend deployment manifest as my queries are timing out, but can't find the correct syntax format expected.
Current Syntax I am ...
0
votes
1
answer
67
views
Creating a log monitor based on multiple lines
I have a bunch of time sensitive functions that I schedule to run asynchronously (using an in-house async job scheduler). I am trying to use my observability tool (datadog) to get alerted if the run ...
0
votes
0
answers
31
views
How to call other master components' /metrics API in a managed environment?
I have a managed version of the Kubernetes cluster, that is, kube-apiserver / kcm / kube-scheduler / etcd is managed, and only kube-apiserver can be directly accessed through the network, and other ...
0
votes
0
answers
12
views
JVM Spring Boot Actuator metrics and JVM memory calculation from Linux [duplicate]
I'm monitoring a Spring Boot application through actuator metrics.
Actuator endpoints reports these metrics:
jvm memory used: 362MB
jvm memory committed: 485MB
jvm memory max: 548MB
Using top linux ...
0
votes
0
answers
32
views
Is it possible to log killed queries reaching threshold max_execution_time
I've set max_execution_time to a sensible value so I can protect MySQL from rogue queries done by users to prevent resource exhaustion.
I need though to find what queries are terminated, to analyse ...
1
vote
0
answers
672
views
Prometheus metrics error "was collected before with the same name and label values"
I have this service made in go that exposes RESTful API. I have a middleware that keeps track of requests via prometheus metrics (countervec and histogram) -> requests_total and request_duration. ...
0
votes
2
answers
363
views
How to secure OpenTelemetry endpoint with Keycloak
How to secure our OpenTelemetry Endpoint with Keycloak?
I am only using OpenTelemetry Collector Gateway ( no agents) in my scenario.
I have created an OTLP/HTTP endpoint and made it publicly ...
0
votes
0
answers
441
views
I am looking to create a custom metric based on BigQuery SQL
The requirement is to monitor the table customer which has a column ingest_timestamp.
I want to send an alert to the user if no data has arrived within last 24 hours. chatGPT has provided me this ...
1
vote
1
answer
206
views
how to create queries in prometheus with wildcard
my application emits counter metrics something like this:
app_counters_some_client_response_code_201_count_value{}
app_counters_some_client_response_code_400_count_value{}
...
0
votes
0
answers
342
views
elastic RUM & APM Traces are not having same trace id
I came across the issue where I am not getting same trace id for RUM & APM transaction.
E.g., I initiate the transaction "add product to basket" and I get the transaction detail in APM (...
2
votes
1
answer
75
views
What's the best practice for restoring the trace id when a 3rd party integration interrupts the trace?
When we receive an HTTP call, we create a new trace id, and pass it to all microservices involved in dealing with that request and the subrequests that result from it.
This works fine until a certain ...
1
vote
1
answer
1k
views
Combine Micrometer's @Observed and B3 Trace ID value
The goal is to combine traces of network calls and spans observed by Micrometer. Network calls are observed with B3-headers. Method calls are observed with @Observed annotation. I have a B3-Trace-ID ...
0
votes
1
answer
520
views
Opentelemetry auto-instrumentation for Node js
I am using ESM in my node application, when I tried opentelemetry to auto instrument my app using @opentelemetry/auto-instrumentations-node package I could not generate any tracing, is this normal ...
0
votes
1
answer
31
views
Can be the refresh interval value in Kibana Stream configured? (version 8.5.0+)
Is it possible to change a refresh/update interval in Kibana 8.5.0 for the live stream functionality in Logs/Stream from the value of 5 seconds?
0
votes
1
answer
43
views
Is counter right datatype to monitor revenue over time in Prometheus?
I am trying to understand which datatype is right choice to monitor revenue metric in Prometheus.
0
votes
1
answer
495
views
Spring's Observability API alternative for @Timed annotation
We're upgrading our app to Spring boot 3 and the new Observability API. In the old way we would just create a timedAspect bean and that would allow us to use @Timed annotations to automatically create ...
3
votes
1
answer
801
views
Open-telemetry auto instrumentation does not work without sidecar
I work at a startup, and we recently migrated our workloads to use Kubernetes, specifically we are running inside a cluster in EKS (AWS).
I'm currently trying to implement a observability stack on our ...
0
votes
1
answer
674
views
td-agent.log doesn't have error logs but the logs are not still appearing in http endpoint
I am sending logs using fluentd to my coralogix account.
I configured everything and made my td-agent.service running properly and without error as shown in the td-agent.log. However, I can't still ...
0
votes
1
answer
228
views
How to monitor Istio stats based on URI using prometheus?
We are using Istio 1.15.4 on EKS 1.23.6.
Our application URL has customer id in it, for eg. https://neeleshgurjar.co.in/api/v1/customer_11
https://neeleshgurjar.co.in/api/v1/customer_12
All requests ...
0
votes
1
answer
676
views
how do u properly use observability and tracing with microservices, using micrometer NOT sleuth?
I'm making a simple microservices project/application with spring boot 3 and you are supposed to now use micrometer for tracing, logging, and monitoring. And guides on this really don't show this ...
0
votes
1
answer
121
views
How to change root activity name in .NET HotChocolate for Elastic APM using a CustomActivityEnricher?
I am trying to change the root activity name in .NET HotChocolate for Elastic APM by creating a custom activity enricher and overriding the CreateRootActivityName method.
public class ...
1
vote
0
answers
100
views
What is the best way to calculate the real and combined memory usage of forked processes?
I am working on a per process memory monitoring (Bash) script but it turns out to be more of a headache than I thought. Especially on forked processes such as PostgreSQL. There are a couple of reasons:...
0
votes
2
answers
771
views
How to add Elastic APM integration from API/CMD/configuration file
I've created a docker-compose file with some configurations that deploy Elasticsearch, Kibana, Elastic Agent all version 8.7.0.
where in the Kibana configuration files I define the police I needed ...