Skip to main content
Filter by
Sorted by
Tagged with
0 votes
0 answers
14 views

Fluentd not inserting logs into MongoDB (using fluent-plugin-mongo)

I'm using Fluentd with the fluent-plugin-mongo plugin to read log files and insert them into MongoDB. The logs are being read and output to stdout as expected, but they are not being inserted into ...
rick's user avatar
  • 751
0 votes
0 answers
8 views

opentelemetry collector, route events to different exporters

We use fluentd-hec to send openshift container logs to Splunk Enterprise and this works nicely. However fluentd-hec is not supported anymore and Splunk offers an alternative, their own distribution of ...
Paul V's user avatar
  • 123
1 vote
1 answer
33 views

In Kibana dashboard, how do you transform a Field value that contains JSON (in fluentd Data View)?

Versions: Elasticsearch / Kibana: 8.6.0 AWS EKS k8s: 1.24 k8s logging operator: ghcr.io/banzaicloud/logging-operator:3.17.10 Our logging stack is Elasticsearch => fluentd => Kibana. We use a ...
Kode Charlie's user avatar
  • 1,395
1 vote
1 answer
19 views

Fluentd 1.16.3 on windows: how to enrich certain log entries without filtering out the rest?

I need to add a custom field to all log entries that contain string "myapp.database - Statement returned". All other logs that do not contain this string, should not be discarded/filtered ...
kamokoba's user avatar
  • 567
2 votes
1 answer
22 views

Fluentd Not able to format json to customized output

Below is the json output which we recieve in Fluentd from OTEL Collector "tags": [ { "key": "otel.library.name", "vStr": "com....
Bhanu Praveen's user avatar
0 votes
1 answer
65 views

fluentd output to s3 failed

I was trying to put logs produced by an application on docker-compose to my s3 bucket using fluentd, but got the following error fluentd | 2024-07-04 09:01:48 +0000 [error]: #0 /usr/lib/...
Garou's user avatar
  • 11
0 votes
3 answers
102 views

Fluentd sending wrong time to cloudwatch

In fluentd, a source is syslog and the target is fluent-plugin-cloudwatch-logs plugin. Everything works, only the time is wrong. <source> @type syslog <transport tcp> </transport&...
David's user avatar
  • 2,962
0 votes
2 answers
87 views

Can I setup multiple fluentd configurations in one fluentd installation?

I want to create multiple pipelines for datastreams that use fluentd. For example, if I have a data stream that comes from kafka, I want to use some filters and then finally send the data to elastic ...
Fouzan's user avatar
  • 19
0 votes
1 answer
85 views

failed to write data into buffer by buffer overflow action=:throw_exception

I have deployed fluentd as kubernetes pod to upload the kubernetes log to AWS OpenSearch. Initially for first few days it was working properly(sending log to AWS OpenSearch), after few days the latest ...
Naveen R's user avatar
0 votes
2 answers
326 views

Docker swarm and fluentd logging driver

I tried to record some logs for front-end nginx-based containers using fluentd docker logging driver, but failed. I ended up with following configuration for fluentd (located in /tmp/fluentd/fluent....
Norritt's user avatar
  • 143
0 votes
0 answers
11 views

How to use the output of a command as input of another command in fluentd?

With fluentd, I want to use each line of the output of jps -lv command (that lists the running JVM on machine), as input argument of jcmd <PID> GC.heap_info (that gives info on heap usage). I've ...
user2655800's user avatar
1 vote
0 answers
24 views

Fluentd plugin how to parse a value with a dynamic key?

Here is the input message= [ { "SNMPv2-MIB::sysUpTime.0"=>"338 days, 20:31:17.37", "SNMPv2-MIB::snmpTrapOID.0"=>"IF-MIB::linkDown&...
user24980214's user avatar
0 votes
0 answers
177 views

Understanding Fluentd Splunk Forwarding Configs

I have set up Fluentd daemonsets to send logs to Splunk from AWS EKS clusters. Logs are being forwarded to HEC endpoint. I however see multiple events appear in a single output line on Splunk ...
Ninja's user avatar
  • 183
1 vote
0 answers
152 views

Fluentd send logs to http

I have a fluentd setup that adds all logs into a new json object called log. I would like to post to http match that only the elements within the log object. How can I do it? Below my configuration: &...
user24903323's user avatar
0 votes
0 answers
12 views

Fluentd - AWS managed opensearch service logging issue

I need configure FluentD to send logs from my AWS-EKS nodes to AWS managed OpenSearch dashboard. FluentD as DaemonSet installed on each of my AWS-EKS node. What is a problem? Problem:- FluentD unable ...
Naveen R's user avatar
0 votes
0 answers
345 views

"buffer space has too many data" BufferOverflowError in fluent

This Fluentd service consumes from Kafka and stores data in OpenSearch. The longest logs are about 32,700 bytes, while typical logs are around 10 to 15 MB. <buffer> chunk_limit_size 50m ...
Kailey's user avatar
  • 1
0 votes
0 answers
78 views

Monitor fluentd.conf with exec plugin

I have fluentd deployed in K8s container. The below is fluentd configuration to monitor the fluentd.conf. If there is any change done to fluentd.conf file the reload.sh script will run which will ...
Dhanu's user avatar
  • 49
1 vote
0 answers
32 views

Add specific log in Docker Container

I have a Java Application (Jboss server) and in the currently VM architecture I have a audit.log configured with log4j <appender name="AUDIT" class="org.jboss.logging.appender....
codi05ro's user avatar
  • 149
0 votes
2 answers
56 views

Troubleshooting Fluentd as a Log Aggregator: Connectivity Issues

I am having trouble sending API logs from one Fluentd server to another, and then to Elasticsearch. When ! tried sending logs directly from one Fluentd server to Elasticsearch, it worked fine. Now, I ...
vidhi yadav's user avatar
0 votes
0 answers
101 views

Fluentd Geoip Plugin is not working in Amazon Linux 2023

While installing 'fluent-gem install fluent-plugin-geo' in Amazon Linux 2023, we get below errors. Is this a known issue already? Note: Fluentd is installed using Repo (Script on website), and all ...
Zaffar Ali's user avatar
1 vote
0 answers
126 views

How to Enable Mutual TLS (mTLS) for Elasticsearch and Fluent Bit?

I'm currently working on securing communication between Elasticsearch and Fluent Bit in my Kubernetes environment. I want to implement Mutual TLS (mTLS) to enhance the security of the communication ...
Md. Nasir Uddin's user avatar
0 votes
2 answers
54 views

How to match optional group in regular expression

I want a regular expression for using in fluentd for parsing nginx error logs. The sample row is: 2024/04/15 09:06:29 [error] 3443790#3443790: *176070165 limiting requests, excess: 2.957 by zone "...
Saeed's user avatar
  • 166
0 votes
0 answers
265 views

Parsing structured log from a container by google cloud ops agent

I have an application running inside a container inside a GCP vm, outputting log that looks like this: {"severity":"INFO","message":"Http request served","...
Haemin Yoo's user avatar
0 votes
1 answer
25 views

LDAP - 389ds - FluentD - Filter plugin - type record_transformer - can it have more than one <record> stanza entries?

Can FluentD 1.16.3/+ have more than one record entries inside filter plugin for type record_transformer? I have the following configuration, but it seems like it's keeping ONLY the last record entry. ...
AKS's user avatar
  • 17.1k
0 votes
0 answers
201 views

FluentD / Fluent-Bit: Concatenate multiple lines of log files and generate one JSON record for all key-value from each line

FluentD 1.16.3 and Fluent-bit 1.8.11 I have the following lines in my container log file /var/log/containers/.log* When a ldap user credentials fail (bad user/password) 024-03-28T16:09:23.048182266Z [...
AKS's user avatar
  • 17.1k
0 votes
0 answers
73 views

unable to serialize JSON type logs In fluentd(logging-operator)

this is my really log { "level": "info", "time": "2024-03-28T10:34:44.345Z", "req": { "id": 6, "method&...
xiao3426's user avatar
0 votes
1 answer
186 views

Filter logs fluent-bit on regex parsing

I have a fluentbit running that scrapes json logs from a dir: [PARSER] Name json Format json Time_Key time # Time_Format %llu Time_Keep On [PARSER] Name ...
Bennimi's user avatar
  • 490
0 votes
0 answers
205 views

Fluentd - Opensearch: Logs not sending to Opensearch

I am new to Fluentd and Opensearch configuration. As part of requirememt , I have to install Opensearch and Fluentd using helmcharts. Using Output plugin configuraton in fluentd, those logs has to ...
Varghese's user avatar
0 votes
1 answer
25 views

how to use fluentd to collect data from a service

I'm new to fluentd. I have a Java-DropWizard.io service running in a docker environment. I can ask this service for a bunch of metrics data in json format by curl http://ip:PORT/admin/metrics. Is ...
Lars's user avatar
  • 3,820
0 votes
1 answer
323 views

Are the fluentbit filter plugin filtering step by step?

Are the fluentbit filters filtering step by step? How it works actually? When the first filter eliminates some data, will the second filter make another elimination from the first filtered data? Could ...
bestwisches's user avatar
0 votes
0 answers
36 views

Azure App Service - unresponsiveness while connecting to Fluentd

I am in the process of pushing the logs Application logs to 'Fluentd' From 'Fluentd' to Dynatrace. The application is Spring Boot application which is deployed as Azure App Service. In the ...
RagaSGNur's user avatar
  • 309
0 votes
0 answers
63 views

Elasticsearch not accepting ILM-Config from Fluentd

I have this Output-Plugin-Config for fluentd: <match logging.**> @type elasticsearch_data_stream host <<fluent-conf-host-elasticsearch>> port <<fluent-conf-port-...
Lutz Buchta's user avatar
0 votes
0 answers
104 views

Can't use Fluentd with the "fluent-plugin-sql" plugin

I want to use fluentd to read data from a database in a dockerized environment. In the container where I have installed fluentd I put this "fluent.conf" file: <source> @type sql ...
nix86's user avatar
  • 2,997
0 votes
0 answers
52 views

How throttling can be prevented across loki , fluentd and fluent-bit

I currently have a logging solution in Kubernetes as follows. Fluent-bit deployed as daemon sets that collect logs from the nodes After collecting logs, fluent-bit forwards data to two destinations -...
ManiPrabu's user avatar
0 votes
0 answers
127 views

export metrics with fluentd

I'm new to Fluentd and recently I've been writing a simple configuration for Fluentd. I send access_log from Nginx to Fluentd with a particular format. Then I parse it in Fluentd and expose metrics to ...
lornejad's user avatar
  • 437
0 votes
1 answer
83 views

Fluentd installation failed with message in Windows

I'm doing testing Fluentd for collecting log files from Apache Tomcat in Windows OS. So, I tried install with [Fluentd-Packages v5.x] and [Calyptia-Fluent v1.3.x] and failed with Windows Message that &...
leeyounsoo's user avatar
0 votes
0 answers
263 views

Remove excess line breaks from s3 log files (fluent-bit s3 output plugin)

I am using fluent-bit s3 output plugin to upload Kubernetes pod logs to s3. I see excessive line breaks in s3 log files as below: 2024-01-24 10:03:34.510 [65b0e07526a14752251fdf7a2e309f58] INFO [Log] ...
Elnur Mammadov's user avatar
0 votes
1 answer
299 views

illegal_argument_exception: index.lifecycle.rollover_alias [alias-efk-stack] does not point to index [efk-stack-pod-name]

Need help for setting up ilm policy for existing indices. I have created first the index Lifecycle Policy which will delete data older than 10 days with name "10-days-archival-policy". Then ...
Mandar Pimplapure's user avatar
1 vote
1 answer
84 views

Transform custom Docker logs into fluentd into elastic search

I will publish some Docker containers incorporating a common logging framework (written in golang). The logging format is a JSON format. There is distinct data in this custom json logging format that ...
ZeroDefect's user avatar
1 vote
1 answer
284 views

Use <source> inside <label @FLUENT_LOG> section in fluentd configuration

I want to change the format of fluentd own logs before sending on stdout. To achieve this, I have captured fluentd logs using label @FLUENT_LOG and then configured a filter to format the logs and then ...
Dhanu's user avatar
  • 49
1 vote
0 answers
43 views

Check with log strout received from fluentd forwarder using fluentd aggregator

In the k8s environment, fluentd is running in daemonset format, so the pod mounted the fluent.conf file as configmap. Below is the contents of the conf file. Collect logs starting with calico-node and ...
심준보's user avatar
1 vote
1 answer
57 views

How could I get fluentd to retain a value from one event then add it to all the following events?

Is there a way to have fluentd retain a value from one event, then append that value to all events from another soruce, until it recieves another similiar event with a new value for that field? For ...
SimonP's user avatar
  • 11
0 votes
0 answers
60 views

How to get metrics for number logs processed in FluentD

Currently, we employ the following approach for sending logs to OpenSearch: receiving application logs in Gzip format stored in S3. We then input this S3 Gzip file(collection of logs) into FluentD, ...
Prakash's user avatar
  • 17
0 votes
0 answers
55 views

CPU Limit for Fluentd agent

I am using Fluentd to tail data from multiple files (one worker for one file). Recently the number of tailed files increased, and so did that of workers. That all resulted in Fluentd's overusing CPU ...
MaiTruongSon's user avatar
0 votes
0 answers
144 views

fluentd k8s how to override tag with rewrite_tag_filter multiple times

I am trying to capture both namespace name and the container name while trying to add them to the tag, using the rewrite_tag_filter My goal: The tag should be: <container_name>.<...
JimShapedCoding's user avatar
0 votes
0 answers
109 views

Fluentd Time Key String Format

Fluentd is listening to the logs of my applications and storing them in MongoDB. Each log has a time key. I want this time key to be stored as a string, but it is being stored as a date type. How can ...
oakkose's user avatar
  • 395
0 votes
0 answers
257 views

how to get the specfic deployment logs with fluent-bit in EKS

I want to upload the specific deployment logs to s3 bucket with fluent-bit. i used below mentioned configuration files: fluent-bit-daemonset.yaml: kind: DaemonSet metadata: name: fluent-bit ...
Muhammad Salahuddin's user avatar
0 votes
0 answers
142 views

Fluentd <Secondary> Functionality Issue

I'm having a problem with a Fluentd configuration that I haven't been able to solve and was wondering if someone might be able to help me or point me in the right direction. I have Fluentd(v1.15.3) ...
user2634876's user avatar
0 votes
0 answers
64 views

Fluentd S3 logs

Trying to push logs from local application to S3 bucket , using fluentd , but the files are created based on index, is it possible to accumulated towards one single file? Any suggestion? Logs are ...
user3557293's user avatar

1
2 3 4 5
27