1,327
questions
0
votes
0
answers
14
views
Fluentd not inserting logs into MongoDB (using fluent-plugin-mongo)
I'm using Fluentd with the fluent-plugin-mongo plugin to read log files and insert them into MongoDB. The logs are being read and output to stdout as expected, but they are not being inserted into ...
0
votes
0
answers
8
views
opentelemetry collector, route events to different exporters
We use fluentd-hec to send openshift container logs to Splunk Enterprise and this works nicely. However fluentd-hec is not supported anymore and Splunk offers an alternative, their own distribution of ...
1
vote
1
answer
33
views
In Kibana dashboard, how do you transform a Field value that contains JSON (in fluentd Data View)?
Versions:
Elasticsearch / Kibana: 8.6.0
AWS EKS k8s: 1.24
k8s logging operator: ghcr.io/banzaicloud/logging-operator:3.17.10
Our logging stack is Elasticsearch => fluentd => Kibana.
We use a ...
1
vote
1
answer
19
views
Fluentd 1.16.3 on windows: how to enrich certain log entries without filtering out the rest?
I need to add a custom field to all log entries that contain string "myapp.database - Statement returned". All other logs that do not contain this string, should not be discarded/filtered ...
2
votes
1
answer
22
views
Fluentd Not able to format json to customized output
Below is the json output which we recieve in Fluentd from OTEL Collector
"tags": [
{
"key": "otel.library.name",
"vStr": "com....
0
votes
1
answer
65
views
fluentd output to s3 failed
I was trying to put logs produced by an application on docker-compose to my s3 bucket using fluentd, but got the following error
fluentd | 2024-07-04 09:01:48 +0000 [error]: #0 /usr/lib/...
0
votes
3
answers
102
views
Fluentd sending wrong time to cloudwatch
In fluentd, a source is syslog and the target is fluent-plugin-cloudwatch-logs plugin. Everything works, only the time is wrong.
<source>
@type syslog
<transport tcp>
</transport&...
0
votes
2
answers
87
views
Can I setup multiple fluentd configurations in one fluentd installation?
I want to create multiple pipelines for datastreams that use fluentd. For example, if I have a data stream that comes from kafka, I want to use some filters and then finally send the data to elastic ...
0
votes
1
answer
85
views
failed to write data into buffer by buffer overflow action=:throw_exception
I have deployed fluentd as kubernetes pod to upload the kubernetes log to AWS OpenSearch.
Initially for first few days it was working properly(sending log to AWS OpenSearch), after few days the latest ...
0
votes
2
answers
326
views
Docker swarm and fluentd logging driver
I tried to record some logs for front-end nginx-based containers using fluentd docker logging driver, but failed.
I ended up with following configuration for fluentd (located in /tmp/fluentd/fluent....
0
votes
0
answers
11
views
How to use the output of a command as input of another command in fluentd?
With fluentd, I want to use each line of the output of jps -lv command (that lists the running JVM on machine), as input argument of jcmd <PID> GC.heap_info (that gives info on heap usage).
I've ...
1
vote
0
answers
24
views
Fluentd plugin how to parse a value with a dynamic key?
Here is the input
message= [
{
"SNMPv2-MIB::sysUpTime.0"=>"338 days, 20:31:17.37",
"SNMPv2-MIB::snmpTrapOID.0"=>"IF-MIB::linkDown&...
0
votes
0
answers
177
views
Understanding Fluentd Splunk Forwarding Configs
I have set up Fluentd daemonsets to send logs to Splunk from AWS EKS clusters. Logs are being forwarded to HEC endpoint. I however see multiple events appear in a single output line on Splunk ...
1
vote
0
answers
152
views
Fluentd send logs to http
I have a fluentd setup that adds all logs into a new json object called log. I would like to post to http match that only the elements within the log object.
How can I do it?
Below my configuration:
&...
0
votes
0
answers
12
views
Fluentd - AWS managed opensearch service logging issue
I need configure FluentD to send logs from my AWS-EKS nodes to AWS managed OpenSearch dashboard.
FluentD as DaemonSet installed on each of my AWS-EKS node.
What is a problem?
Problem:-
FluentD unable ...
0
votes
0
answers
345
views
"buffer space has too many data" BufferOverflowError in fluent
This Fluentd service consumes from Kafka and stores data in OpenSearch.
The longest logs are about 32,700 bytes, while typical logs are around 10 to 15 MB.
<buffer>
chunk_limit_size 50m
...
0
votes
0
answers
78
views
Monitor fluentd.conf with exec plugin
I have fluentd deployed in K8s container. The below is fluentd configuration to monitor the fluentd.conf. If there is any change done to fluentd.conf file the reload.sh script will run which will ...
1
vote
0
answers
32
views
Add specific log in Docker Container
I have a Java Application (Jboss server) and in the currently VM architecture I have a audit.log configured with log4j
<appender name="AUDIT" class="org.jboss.logging.appender....
0
votes
2
answers
56
views
Troubleshooting Fluentd as a Log Aggregator: Connectivity Issues
I am having trouble sending API logs from one Fluentd server to another, and then to Elasticsearch. When ! tried sending logs directly from one Fluentd server to Elasticsearch, it worked fine. Now, I ...
0
votes
0
answers
101
views
Fluentd Geoip Plugin is not working in Amazon Linux 2023
While installing 'fluent-gem install fluent-plugin-geo' in Amazon Linux 2023, we get below errors. Is this a known issue already?
Note: Fluentd is installed using Repo (Script on website), and all ...
1
vote
0
answers
126
views
How to Enable Mutual TLS (mTLS) for Elasticsearch and Fluent Bit?
I'm currently working on securing communication between Elasticsearch and Fluent Bit in my Kubernetes environment. I want to implement Mutual TLS (mTLS) to enhance the security of the communication ...
0
votes
2
answers
54
views
How to match optional group in regular expression
I want a regular expression for using in fluentd for parsing nginx error logs.
The sample row is:
2024/04/15 09:06:29 [error] 3443790#3443790: *176070165 limiting requests, excess: 2.957 by zone "...
0
votes
0
answers
265
views
Parsing structured log from a container by google cloud ops agent
I have an application running inside a container inside a GCP vm, outputting log that looks like this: {"severity":"INFO","message":"Http request served","...
0
votes
1
answer
25
views
LDAP - 389ds - FluentD - Filter plugin - type record_transformer - can it have more than one <record> stanza entries?
Can FluentD 1.16.3/+ have more than one record entries inside filter plugin for type record_transformer?
I have the following configuration, but it seems like it's keeping ONLY the last record entry.
...
0
votes
0
answers
201
views
FluentD / Fluent-Bit: Concatenate multiple lines of log files and generate one JSON record for all key-value from each line
FluentD 1.16.3 and Fluent-bit 1.8.11
I have the following lines in my container log file /var/log/containers/.log*
When a ldap user credentials fail (bad user/password)
024-03-28T16:09:23.048182266Z [...
0
votes
0
answers
73
views
unable to serialize JSON type logs In fluentd(logging-operator)
this is my really log
{
"level": "info",
"time": "2024-03-28T10:34:44.345Z",
"req": {
"id": 6,
"method&...
0
votes
1
answer
186
views
Filter logs fluent-bit on regex parsing
I have a fluentbit running that scrapes json logs from a dir:
[PARSER]
Name json
Format json
Time_Key time
# Time_Format %llu
Time_Keep On
[PARSER]
Name ...
0
votes
0
answers
205
views
Fluentd - Opensearch: Logs not sending to Opensearch
I am new to Fluentd and Opensearch configuration.
As part of requirememt , I have to install Opensearch and Fluentd using helmcharts. Using Output plugin configuraton in fluentd, those logs has to ...
0
votes
1
answer
25
views
how to use fluentd to collect data from a service
I'm new to fluentd.
I have a Java-DropWizard.io service running in a docker environment.
I can ask this service for a bunch of metrics data in json format by curl http://ip:PORT/admin/metrics.
Is ...
0
votes
1
answer
323
views
Are the fluentbit filter plugin filtering step by step?
Are the fluentbit filters filtering step by step? How it works actually?
When the first filter eliminates some data, will the second filter make another elimination from the first filtered data?
Could ...
0
votes
0
answers
36
views
Azure App Service - unresponsiveness while connecting to Fluentd
I am in the process of pushing the logs
Application logs to 'Fluentd'
From 'Fluentd' to Dynatrace.
The application is Spring Boot application which is deployed as Azure App Service.
In the ...
0
votes
0
answers
63
views
Elasticsearch not accepting ILM-Config from Fluentd
I have this Output-Plugin-Config for fluentd:
<match logging.**>
@type elasticsearch_data_stream
host <<fluent-conf-host-elasticsearch>>
port <<fluent-conf-port-...
0
votes
0
answers
104
views
Can't use Fluentd with the "fluent-plugin-sql" plugin
I want to use fluentd to read data from a database in a dockerized environment.
In the container where I have installed fluentd I put this "fluent.conf" file:
<source>
@type sql
...
0
votes
0
answers
52
views
How throttling can be prevented across loki , fluentd and fluent-bit
I currently have a logging solution in Kubernetes as follows.
Fluent-bit deployed as daemon sets that collect logs from the nodes
After collecting logs, fluent-bit forwards data to two destinations
-...
0
votes
0
answers
127
views
export metrics with fluentd
I'm new to Fluentd and recently I've been writing a simple configuration for Fluentd. I send access_log from Nginx to Fluentd with a particular format. Then I parse it in Fluentd and expose metrics to ...
0
votes
1
answer
83
views
Fluentd installation failed with message in Windows
I'm doing testing Fluentd for collecting log files from Apache Tomcat in Windows OS.
So, I tried install with [Fluentd-Packages v5.x] and [Calyptia-Fluent v1.3.x] and failed with Windows Message that &...
0
votes
0
answers
263
views
Remove excess line breaks from s3 log files (fluent-bit s3 output plugin)
I am using fluent-bit s3 output plugin to upload Kubernetes pod logs to s3. I see excessive line breaks in s3 log files as below:
2024-01-24 10:03:34.510 [65b0e07526a14752251fdf7a2e309f58] INFO [Log] ...
0
votes
1
answer
299
views
illegal_argument_exception: index.lifecycle.rollover_alias [alias-efk-stack] does not point to index [efk-stack-pod-name]
Need help for setting up ilm policy for existing indices. I have created first the index Lifecycle Policy which will delete data older than 10 days with name "10-days-archival-policy". Then ...
1
vote
1
answer
84
views
Transform custom Docker logs into fluentd into elastic search
I will publish some Docker containers incorporating a common logging framework (written in golang). The logging format is a JSON format.
There is distinct data in this custom json logging format that ...
1
vote
1
answer
284
views
Use <source> inside <label @FLUENT_LOG> section in fluentd configuration
I want to change the format of fluentd own logs before sending on stdout. To achieve this, I have captured fluentd logs using label @FLUENT_LOG and then configured a filter to format the logs and then ...
1
vote
0
answers
43
views
Check with log strout received from fluentd forwarder using fluentd aggregator
In the k8s environment, fluentd is running in daemonset format, so the pod mounted the fluent.conf file as configmap. Below is the contents of the conf file. Collect logs starting with calico-node and ...
1
vote
1
answer
57
views
How could I get fluentd to retain a value from one event then add it to all the following events?
Is there a way to have fluentd retain a value from one event, then append that value to all events from another soruce, until it recieves another similiar event with a new value for that field?
For ...
0
votes
0
answers
60
views
How to get metrics for number logs processed in FluentD
Currently, we employ the following approach for sending logs to OpenSearch: receiving application logs in Gzip format stored in S3. We then input this S3 Gzip file(collection of logs) into FluentD, ...
0
votes
0
answers
55
views
CPU Limit for Fluentd agent
I am using Fluentd to tail data from multiple files (one worker for one file). Recently the number of tailed files increased, and so did that of workers. That all resulted in Fluentd's overusing CPU ...
0
votes
0
answers
144
views
fluentd k8s how to override tag with rewrite_tag_filter multiple times
I am trying to capture both namespace name and the container name while trying to add them to the tag, using the rewrite_tag_filter
My goal: The tag should be: <container_name>.<...
0
votes
0
answers
109
views
Fluentd Time Key String Format
Fluentd is listening to the logs of my applications and storing them in MongoDB. Each log has a time key. I want this time key to be stored as a string, but it is being stored as a date type. How can ...
0
votes
0
answers
257
views
how to get the specfic deployment logs with fluent-bit in EKS
I want to upload the specific deployment logs to s3 bucket with fluent-bit.
i used below mentioned configuration files:
fluent-bit-daemonset.yaml:
kind: DaemonSet
metadata:
name: fluent-bit
...
0
votes
0
answers
142
views
Fluentd <Secondary> Functionality Issue
I'm having a problem with a Fluentd configuration that I haven't been able to solve and was wondering if someone might be able to help me or point me in the right direction.
I have Fluentd(v1.15.3) ...
0
votes
0
answers
64
views
Fluentd S3 logs
Trying to push logs from local application to S3 bucket , using fluentd , but the files are created based on index, is it possible to accumulated towards one single file? Any suggestion?
Logs are ...