Questions tagged [observability]

monitoring the internal state of a system by looking at its output

Filter by
Sorted by
Tagged with
2 votes
0 answers
197 views

Exception in observability stack after upgrading Spring Boot from 3.1.3 to 3.2.0 using opentelemetry-java

Problem: When upgrading the observability stack from Spring Boot version 3.1.3 to 3.2.0 using OTel Java SDK, the following exception is encountered: 2023-11-26T16:00:43.087Z WARN [order service,,] 1 -...
kcsurapaneni's user avatar
0 votes
0 answers
14 views

Observing subprocess execution times and memory usages, should it be a Metric? or an Event?

I have an application that runs various subprocesses dynamically during its runtime. I aim to create graphs depicting the execution time of these subprocesses and their corresponding memory usage. ...
Meir Tolpin's user avatar
0 votes
1 answer
97 views

Collecting data with Dynatrace OneAgent or OpenTelemetry

I am currently thinking about debating using OpenTelemetry to collect data and then send it to Dynatrace or collecting data with the OneAgent. Have you ever made experiences with this? My question is, ...
Max Julius's user avatar
-1 votes
1 answer
35 views

Using ELK in Azure Cloud

We are using Azure App Insights for all tracing . monitoring and log requirements of our applications running in Azure. We have seen cases where teams ship the logs to ELK for observability. We do not ...
lives's user avatar
  • 1,215
1 vote
0 answers
41 views

How to get request/response body with uprobe?

https://github.com/open-telemetry/opentelemetry-go-instrumentation internal/pkg/instrumentation/bpf/net/http/server/probe.go It injects a lot of fields,but Event only contains part of. How can I get ...
Evan's user avatar
  • 11
0 votes
1 answer
43 views

I got an alert for a latency spike in my Cloud Spanner database. How can I debug it?

I am using Cloud Spanner as the database for my application. For certain queries the latency has spiked. I would like to know how can I debug it? I tried looking at the system insights page but couldn'...
Naren Mehra's user avatar
0 votes
1 answer
99 views

fluentd vs prometheus vs elasticsearch, when to use what?

When to use fluentd vs prometheus vs elasticsearch? There seems to be overlapping between these tools interms of the feature set they provide but I have seen projects using all these 3 tools in unison....
pingpong2020's user avatar
0 votes
1 answer
30 views

How to make a Ballerina function observable?

In Ballerina, the main, resource, and remote functions come inherently equipped with observability to enable metrics and tracing monitoring. However, I wish to extend this observability to some of the ...
Gabil's user avatar
  • 56
2 votes
1 answer
70 views

Parse string of key=values with escaped characters

Loki outputs the following log in a key-value format with the structure key1=value1 key2=value2: level=info ts=2023-10-20T14:30:48.716410806Z caller=metrics.go:159 component=frontend org_id=fake ...
Camões's user avatar
  • 81
0 votes
1 answer
52 views

Spring Observability questions

I implemented code from this article: https://programmingtechie.com/2023/09/09/spring-boot3-observability-grafana-stack/ Can I enable tracing only for specific endpoints ? Trace id between services ...
Bondar Anton's user avatar
1 vote
1 answer
50 views

OpenTelemetry adjust sampling ratio programmatically

In the OpenTelemetry SDK for JS, is there a way to modify the TraceIdRatioBasedSampler sampling ratio at run time? Alternatively, I was considering creating a new sampler and updating the trace ...
davids's user avatar
  • 6,291
2 votes
0 answers
71 views

Does wekbit have an equivalent performance API to chrome for measuring memory usage?

Chrome has measureUserAgentSpecificMemory, which "is used to estimate the memory usage of a web application including all its iframes and workers." Is there an equivalent for Safari? The ...
alleydustin's user avatar
0 votes
0 answers
74 views

How to write query in PromQL and assign it to a variable for alert conditions?

I have a below query from SignalFx and i need to convert it to PromQL: A = data('signal_apiserver_graphql.errors_count', filter=filter('kubernetes_cluster', 'dev-001', 'dev-003', 'dev-002', 'dev-000'...
Witty Apps's user avatar
0 votes
0 answers
25 views

How to integrate AWS X-ray with this format StepFunction => EventBridge => Downstream services

How to include EventBridge Triggered Downstream Services in X-ray Service Map? I have a step function with tracing enabled, and within this step function, there's an EventBridge integration that both ...
YulePale's user avatar
  • 6,944
0 votes
0 answers
20 views

Unable to export spans from OTEL to Elastic APM Server

I have a nestjs application that i want to instrument. I am collecting traces using OTEL and exporting it to zipking and elastic apm server. My main focus is to trace all my graphql queries. I am able ...
Hammad's user avatar
  • 1
0 votes
0 answers
7 views

How can I deliver custom micrometer configuration for use by corporate internal microservices?

I want our internal corporate microservices to use recommended Observability configuration merely by adding in the build dependency. One way to do this is with Spring Boot starter. Assume the teams ...
JJFraney's user avatar
0 votes
0 answers
25 views

Observability for DLQs? [AWS SQS]

I'm trying to understand what is the best way to implement some sort of Observability for DLQs. I know it's possible to have some monitoring (e.g.: There are 3 messages in the DLQ XYZ) I'm trying to ...
jonathanlima's user avatar
0 votes
1 answer
33 views

unknown field `healthcheck` with Vector

I am trying to expose the endpoint "http://localhost:8686/health" using Vector so that I can monitor the Vector's health. After research, I came across guides that recommend putting the ...
Stanley Ulili's user avatar
0 votes
1 answer
138 views

Unable to send open telemetry traces to OpenSearch

I am trying to send open telemetry traces to OpenSearch but I am unable to. The traces are visible on Jeager UI but they are not getting ingested in OpenSearch .I have tried everything ,created ...
Madiha Samad's user avatar
0 votes
0 answers
39 views

How to export logs for certain interval from Kibana Observability/Logs/Stream?

We need to export logs from the stream for a specific time interval (which is set on the top of the main stream frame) into JSON, CSV or any other human-readable format. The logs "live" 7 ...
Arsenii's user avatar
  • 685
0 votes
1 answer
37 views

DataDog alert includes filtered out environment tag

I have a DataDog query alert like so: min(last_5m):sum:my_service.verification.failure{service:"my_service" AND env:"production"} by {region,env} >= 1 And I enabled the ...
DLL's user avatar
  • 11
0 votes
0 answers
55 views

Why use Open Telemetry Logging in Kubernetes dotnet service?

I have a dotnet web api that has been deployed to a Kubernetes cluster. I see a lot of experienced folks suggesting using Open Telemetry SDK with Console exporter instead of dotnet inbuilt logging ...
Abhijit's user avatar
  • 175
0 votes
0 answers
17 views

Spring boot Observablity for Event lifecycle

I am creating a application with spring boot 3 and its observability feature using micrometer and actuators. What my application does is, it receives a alert, generates event based on this alert, ...
Dheeraj Kukreja's user avatar
0 votes
1 answer
686 views

Axios HTTP backend call generates Opentelemetry span that is not correlated with the parent span

I'm trying to perform manual instrumentation in NodeJs (v18.16.1) using opentelemetry. The scenario is very simple, I'm making an axios HTTP call to a dummy JSON API and I want to collect the ...
Liviu Gheorghe's user avatar
1 vote
1 answer
1k views

Error Loki distributed with Helm: no private IP address found, and explicit IP not provided

I'm configuring loki-distributed on a kubernetes cluster via helm, but I'm getting the following error: failed to create memberlist: Failed to get final advertise address: no private IP address found, ...
SrCabra's user avatar
  • 33
1 vote
1 answer
388 views

Dependencies do not show with OpenTelemetry automatic instrumentation for PHP

I have a very simple stack of PHP + Otel auto instrumetation + Elastic APM + Prometheus. My stack gets tracing and metrics information from my php app. My tracings are forwarded to the elastic cloud, ...
Pablo Marques's user avatar
0 votes
0 answers
26 views

Need to integrate Splunk with Git such that Splunk can clone and commit to Git

If Splunk detects certain events, it needs to trigger cloning, committing, and creating a pull request on Git. I can find add-ons on Splunkbase that let me send data from Git to Splunk, but is there ...
Komal Agarwal's user avatar
0 votes
1 answer
141 views

How do i set response_header_timeout in Thanos Query Frontend Manifest?

I want to increase the response_header_timeout in thanos query-frontend deployment manifest as my queries are timing out, but can't find the correct syntax format expected. Current Syntax I am ...
Shrey Makwana's user avatar
0 votes
1 answer
67 views

Creating a log monitor based on multiple lines

I have a bunch of time sensitive functions that I schedule to run asynchronously (using an in-house async job scheduler). I am trying to use my observability tool (datadog) to get alerted if the run ...
shridharama's user avatar
0 votes
0 answers
31 views

How to call other master components' /metrics API in a managed environment?

I have a managed version of the Kubernetes cluster, that is, kube-apiserver / kcm / kube-scheduler / etcd is managed, and only kube-apiserver can be directly accessed through the network, and other ...
flyer's user avatar
  • 9,400
0 votes
0 answers
12 views

JVM Spring Boot Actuator metrics and JVM memory calculation from Linux [duplicate]

I'm monitoring a Spring Boot application through actuator metrics. Actuator endpoints reports these metrics: jvm memory used: 362MB jvm memory committed: 485MB jvm memory max: 548MB Using top linux ...
Michel Foucault's user avatar
0 votes
0 answers
32 views

Is it possible to log killed queries reaching threshold max_execution_time

I've set max_execution_time to a sensible value so I can protect MySQL from rogue queries done by users to prevent resource exhaustion. I need though to find what queries are terminated, to analyse ...
Baptiste Mille-Mathias's user avatar
1 vote
0 answers
672 views

Prometheus metrics error "was collected before with the same name and label values"

I have this service made in go that exposes RESTful API. I have a middleware that keeps track of requests via prometheus metrics (countervec and histogram) -> requests_total and request_duration. ...
rolldawg's user avatar
0 votes
2 answers
363 views

How to secure OpenTelemetry endpoint with Keycloak

How to secure our OpenTelemetry Endpoint with Keycloak? I am only using OpenTelemetry Collector Gateway ( no agents) in my scenario. I have created an OTLP/HTTP endpoint and made it publicly ...
Vaibhav Shah's user avatar
0 votes
0 answers
441 views

I am looking to create a custom metric based on BigQuery SQL

The requirement is to monitor the table customer which has a column ingest_timestamp. I want to send an alert to the user if no data has arrived within last 24 hours. chatGPT has provided me this ...
Dinesh's user avatar
  • 319
1 vote
1 answer
206 views

how to create queries in prometheus with wildcard

my application emits counter metrics something like this: app_counters_some_client_response_code_201_count_value{} app_counters_some_client_response_code_400_count_value{} ...
Shahbaz Khan's user avatar
0 votes
0 answers
342 views

elastic RUM & APM Traces are not having same trace id

I came across the issue where I am not getting same trace id for RUM & APM transaction. E.g., I initiate the transaction "add product to basket" and I get the transaction detail in APM (...
pratiksha tiwari's user avatar
2 votes
1 answer
75 views

What's the best practice for restoring the trace id when a 3rd party integration interrupts the trace?

When we receive an HTTP call, we create a new trace id, and pass it to all microservices involved in dealing with that request and the subrequests that result from it. This works fine until a certain ...
Federico Fissore's user avatar
1 vote
1 answer
1k views

Combine Micrometer's @Observed and B3 Trace ID value

The goal is to combine traces of network calls and spans observed by Micrometer. Network calls are observed with B3-headers. Method calls are observed with @Observed annotation. I have a B3-Trace-ID ...
Dmytro Nasyrov's user avatar
0 votes
1 answer
520 views

Opentelemetry auto-instrumentation for Node js

I am using ESM in my node application, when I tried opentelemetry to auto instrument my app using @opentelemetry/auto-instrumentations-node package I could not generate any tracing, is this normal ...
parv bhardwaj's user avatar
0 votes
1 answer
31 views

Can be the refresh interval value in Kibana Stream configured? (version 8.5.0+)

Is it possible to change a refresh/update interval in Kibana 8.5.0 for the live stream functionality in Logs/Stream from the value of 5 seconds?
ps-aux's user avatar
  • 11.7k
0 votes
1 answer
43 views

Is counter right datatype to monitor revenue over time in Prometheus?

I am trying to understand which datatype is right choice to monitor revenue metric in Prometheus.
Datta Diware's user avatar
0 votes
1 answer
495 views

Spring's Observability API alternative for @Timed annotation

We're upgrading our app to Spring boot 3 and the new Observability API. In the old way we would just create a timedAspect bean and that would allow us to use @Timed annotations to automatically create ...
nihalg's user avatar
  • 3
3 votes
1 answer
801 views

Open-telemetry auto instrumentation does not work without sidecar

I work at a startup, and we recently migrated our workloads to use Kubernetes, specifically we are running inside a cluster in EKS (AWS). I'm currently trying to implement a observability stack on our ...
Victor Chacon's user avatar
0 votes
1 answer
674 views

td-agent.log doesn't have error logs but the logs are not still appearing in http endpoint

I am sending logs using fluentd to my coralogix account. I configured everything and made my td-agent.service running properly and without error as shown in the td-agent.log. However, I can't still ...
EngineerDegz's user avatar
0 votes
1 answer
228 views

How to monitor Istio stats based on URI using prometheus?

We are using Istio 1.15.4 on EKS 1.23.6. Our application URL has customer id in it, for eg. https://neeleshgurjar.co.in/api/v1/customer_11 https://neeleshgurjar.co.in/api/v1/customer_12 All requests ...
Neelesh Gurjar's user avatar
0 votes
1 answer
676 views

how do u properly use observability and tracing with microservices, using micrometer NOT sleuth?

I'm making a simple microservices project/application with spring boot 3 and you are supposed to now use micrometer for tracing, logging, and monitoring. And guides on this really don't show this ...
java beginner's user avatar
0 votes
1 answer
121 views

How to change root activity name in .NET HotChocolate for Elastic APM using a CustomActivityEnricher?

I am trying to change the root activity name in .NET HotChocolate for Elastic APM by creating a custom activity enricher and overriding the CreateRootActivityName method. public class ...
Y G's user avatar
  • 13
1 vote
0 answers
100 views

What is the best way to calculate the real and combined memory usage of forked processes?

I am working on a per process memory monitoring (Bash) script but it turns out to be more of a headache than I thought. Especially on forked processes such as PostgreSQL. There are a couple of reasons:...
Claudio Kuenzler's user avatar
0 votes
2 answers
771 views

How to add Elastic APM integration from API/CMD/configuration file

I've created a docker-compose file with some configurations that deploy Elasticsearch, Kibana, Elastic Agent all version 8.7.0. where in the Kibana configuration files I define the police I needed ...
dpfdg dfsg's user avatar