Questions tagged [fluentd]
Fluentd is open-source and distributed data collector, which receives logs in JSON format, buffers them, and sends them to other systems like Amazon S3, MongoDB, Hadoop, Loki(Grafana system) or other Fluentds.
1,303
questions
0
votes
0
answers
7
views
Is there a way to conditionally configure a source input plugin in fluentd?
I have a generic fluentd conf that I am using for multiple services. The S3 source input plugin is specific to one service and I want to know if there is a way to conditionally inject this into ...
0
votes
1
answer
16
views
Is the Splunk Universal Forwarder capable of sending data to FluentD
I have multiple Splunk Universal Forwarders, and I am looking to send them to an ETL tool. I am trying to evaluate whether FluentD is a compatible option for this.
FluentD has built in plug-ins for ...
0
votes
1
answer
46
views
access list item by Fluentd
I work on a yaml file configuration of Fluentd. Input data comes from kafka as json file.
some of them are lists lik:
{ a:[ a1:"1", a2:"2", ... ], b:"10" }
How can ...
0
votes
0
answers
31
views
Using Fluentd, how to send logs from specific K8S pods to specific domains and indexes in OpenSearch?
I have added podLabels to the Helm Charts of the Deployments with the exact domain and index I want them in:
podLabels:
tenant: <tenant-name>
domain: <domain-endpoint>
...
0
votes
0
answers
20
views
Error Elasticsearch client not compatible with Elasticsearch server?
I have installed td-agent and I am trying to upload data to Elasticsearch. Below is the td-agent.conf file:
<source>
@type tail
path /home/rocket/PycharmProjects/EFK/log.json
pos_file /...
0
votes
1
answer
38
views
Is there any way to implement zstd compression in fluentd s3 plugin?
We have the requirement to compress and send files to S3 with zstd compression. I am trying to implement zstandard compression plugin with fluentd s3 plugin so that the data that is sent to s3 using ...
-2
votes
1
answer
45
views
Index not found exception while sending data to Opensearch AWS using td agent
I have set up Opensearch in AWS. I have installed td-agent in Ubuntu 18.04. Below is my td-agent.conf file:
<source>
@type tail
path /home/rocket/PycharmProjects/EFK/log.json
pos_file /...
0
votes
0
answers
31
views
Fluentd JSON string parsing with multiple data type in array
I am trying to set up a logging pipeline with Fluentd and elastic search. One of my log patterns looks like the following:
{
"key": "value",
"inputs": [
[
&...
1
vote
0
answers
31
views
Decode stream data from firehose and send to HTTP endpoint
I have setup a Kinesis firehose to stream MSK logs to a http endpoint. The http endpoint is a log collector (fluentd) application. The data I receive in the fluentd app is base64 encoded, I want to ...
0
votes
0
answers
24
views
How to handle multiline and single line logs with one Fluend and Elasticsearch
In my cluster I have set up Fluentd DaemonSet for Elasticsearch. Its working fine and I can see logs in Kibana. We have different pods/containers. Some are Java, Python, Nginx, etc.
The problem at the ...
0
votes
0
answers
31
views
Unable to use snappy compression with fluentdv1.x
Unable to use snappy compression with fluentdv1.x, getting an exception when the producer pushes messages. In fluent configuration, "snappy" compression is enabled.
Using docker official ...
0
votes
0
answers
23
views
Send Cloudwatch logs to Loki
I have a k8s cluster where I installed Loki to keep track of all the Pods' logs. I also have an AWS API gateway that sends API requests to a service in my cluster.
I also enable Cloudwatch logs for ...
0
votes
0
answers
28
views
Gem Permission Error When installing fluentd plugin
When running docker on Jenkins i am getting the following error:
While executing gem ... (Gem::FilePermissionError)
You don't have write permissions for the /usr/lib/ruby/gems/3.1.0 directory.
[0m[...
2
votes
0
answers
23
views
fluentd kafka frequent connection closes - Kafka High cpu
We writing from around 1000 ec2 machines to AWS MSK using kafka fluentd plugin. They write continuously throughout the day.
The aws.kafka.connection_creation_rate metric graph and the aws.kafka....
0
votes
1
answer
22
views
ElasticSearch as a source for FluentD
Is there any way to setup ElasticSearch as a source for Fluentd?
All documentations describes how to set ES as a destination but I have some data which has to be processed by Fluentd and delivered to ...
0
votes
0
answers
20
views
parser hash_value_field replaces existing field instead of adding to it
I'd like to parse a JSON message as output by Kubernetes and add the result into an existing kubernetes key instead of replacing it.
Example of what I want:
# input data: {"message":{"...
0
votes
0
answers
35
views
Exclude logs from fluentd using exclude directive not working
Trying to exclude logs using the grep's exclude directive.
<filter kubernetes.var.log.containers.**>
@type grep
<exclude>
key kubernetes.pod_name
pattern /^podname-*/
</...
0
votes
2
answers
55
views
Fluentd: - problem with regex while parsing log
I have this fluentd configuration:
<source>
@type tail
<parse>
@type regexp
expression /^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*...
0
votes
0
answers
32
views
Cannot fetch hosts : [index_not_found_exception] no such index [kibana_sample_data_logs]
I am doing an implementain of EFK(elasticsearch, fluentd, kibana) stack on local host using docker container. I created a seprate nginx container to generate logs. I have also added the logtrail ...
0
votes
0
answers
45
views
Cannot send logs to Kibana via ELK Stack
Hello Stackoverflow community,
I have AWS EKS and I would like to install the ELK stack (v8.5.0) in order to leverage logging for my containers.
I am using HelmChart for it, and for Elasticsearch I am ...
0
votes
1
answer
54
views
EFK missing geo_point
I am running EFK using ECK 8.5.3. fluentd ConfigMap:
@type geoip
# Specify one or more geoip lookup field which has ip address (default: host)
geoip_lookup_keys IP
# Specify ...
0
votes
0
answers
16
views
How to parse my string o/p of logs to JSON in fluentd
log": "2022-12-15 09:52:52 +0000 [info]: #0 disable filter chain optimization because [Fluent::Plugin::KubernetesMetadataFilter, Fluent::Plugin::RecordTransformerFilter] uses #filter_stream ...
0
votes
0
answers
27
views
Why is "log_source" field blank in logs sent to Splunk by fluentd?
Dashboards show zero log Volume in the Jfrog App for Splunk. Looking at the query in Search pane, they contain "log_source" field which is consistently blank. What field should be used and ...
0
votes
1
answer
117
views
Fluentd record transformer conditional statements to add key if not present
I want to add a message field to the log if it is not present in the logs.
Here's the relevant fluentd configuration:
<filter **>
@type record_transformer
enable_ruby true
<record>
...
1
vote
0
answers
33
views
Why does FluentD in_forward plugin treat JSON object as an Array and log Incoming chunk is broken?
The code below is executed from a FluentD node in AWS. FluentD warns of Incoming chunk is broken here. The TLS connection is successful. I am using ruby 2.7.0p0 (2019-12-25 revision 647ee6f091) [...
0
votes
0
answers
29
views
td-agent configuration for postgresql Logs
I have installed the fluentd agent in my VM where postgresql is installed. Now I am looking for the td-agent.conf file which could help me to ship the pg logs and /var/log/messages to kibana. I am ...
0
votes
1
answer
42
views
Issue while Installing calyptia-fluentd on RHEL7
I'm trying to install RPM Package on the RHEL7 following the installation steps guided on this page:
I was able to install the td-agent successfully, launched the daemon, and tested it by sending ...
0
votes
0
answers
47
views
Fluentd installation - Centos
I am trying to install Fluentd software on Centos servers and I get the error below:
Building native extensions. This could take a while...
ERROR: Error installing fluentd:
ERROR: Failed to ...
0
votes
0
answers
86
views
How do I make fluent-bit forward docker logs?
I am trying to run a fluent-bit container according to https://www.velebit.ai/blog/tech-blog-collecting-logs-in-docker-clusters/. Note the ports section in the docker-compose.yml. I have td-agent ...
0
votes
1
answer
24
views
Fluentd SQL output with complex JSON object failing with NULL value
I'm trying to build a fluentd pipeline that pulls data from an API and puts it in PostgresQL:
<source>
@type http_pull
tag energydata
url http://10.0.0.30:8080/data
interval 10s
...
1
vote
0
answers
67
views
Fluentd Not able to scrape high volume logs
I am trying to Use fluentd to push logs to loki using fluentd-loki plugin. I am not able to make fluentd ingest logs in realtime if the logs exceed 24000 lines/sec. I need help in configuration of ...
0
votes
0
answers
35
views
How to unnest/flatten json in fluentd using parser filter?
This is my current config.
<source>
@type dummy
rate 1
tag eggsample
dummy {"event":"signup","context":{"ip":"105.175.82.28"}}...
1
vote
0
answers
121
views
Fluent-bit parsing json in side of a string
I have a k8s logging stack set up like this:
fluent-bit => fluentd => elastic
I have a working set up with this config:
fluent-bit.conf
[SERVICE]
Daemon Off
Parsers_File custom_parsers.conf
...
0
votes
0
answers
17
views
toggle specific plugin in FluentD (through k8s manifest file)
I have an EFK stack that outputs EKS cluster logs to both Elasticsearch and S3. I wonder if there's a way to add a switch to enable/disable outputs to S3 .. maybe using an ENV in the FluentD manifest ...
0
votes
0
answers
30
views
fluentD not parisng logs without a timestap
This is my current fluentID configuration that parses JSON and standard logs with a timestamp:
<source>
@id fluentd-containers.log
@type tail
path /var/log/containers/*.log
...
0
votes
0
answers
69
views
How to index the data received from OpenShift cluster log forwarder
I am receiving the data to my Elasticsearch from Openshift log forwarder (Fluentd) directly without any pipeline or integrations. I am using an open-source Elasticsearch which is outside of the ...
1
vote
0
answers
58
views
need a fluentd sample to collect data from a url and send it to a rabbitmq container
Im new to fluentd, im having a fluentd container with fluent.conf
<source>
@type http
port 9090
bind my-ip-addr
tag api
</source>
<match api>
@type rabbitmq
host ...
0
votes
0
answers
30
views
Large number of open files Linux on Fluentd machine
I have several questions regarding a Ubuntu machine which I am working on.
First of all here is the output of max open files:
$ cat /proc/sys/fs/file-max
1024000
However when checking how many are ...
0
votes
1
answer
90
views
Fluentbit pump local file to splunk
I am trying to pump local file to splunk using fluentbit. The Splunk is currently https and secure.
I kept encountering error message of unexpected EOF, I am not sure what have I done wrongly in the ...
1
vote
1
answer
53
views
Could not communicate to Elasticsearch
I am trying to send my node app logs to fluentd to elasticsearch to kibana, but having a problem connecting fluentd with elasticsearch with docker. I want to dockerize this efk stack.
I have attached ...
0
votes
0
answers
87
views
sending data on the basis of json key value pair in fluentbit output
wants to send JSON log data that contains a key-value pair. I want to send them to different files based on the key-value pair
ex:
I am having below the data
1. {"ABC": true, "BDC"...
0
votes
0
answers
40
views
How to use fluentd to parse a log file
all.
I have trouble to understand the logic of fluentd usage.
I want to parse a local log file. I have defined config file in yml.
System is Windows 10.
I think td would help me as following.
td -c ...
0
votes
1
answer
111
views
Fluentd doesn't parse json log from docker
I have some problems parsing json logs that were received from docker container. I know this question is probably a duplicate but none of the solutions found, including the documentation (https://docs....
0
votes
0
answers
99
views
FluentD Output in Plain Text (non-json) format
I'm new to FluentD and I'm trying to determine if we can replace our current syslog application with FluentD. The issue that I'm trying to solve is compatability between FluentD and Legacy Application ...
0
votes
0
answers
48
views
Rename a field based on the value of another field in fluentd
I have a log looks like this:
{
eventname: "AAA.amazon.com",
requestParameter: {...}
...
}
I want to rename requestParameter based on a regex of the value of eventname field in Fluentd.
It ...
0
votes
0
answers
84
views
Fluentd tail reads every time from the start of the file
I am trying to forward syslogs,
I start the td-agent with read_from_head param set to false.
So on checking logs nothing happens, then I added a line to the target file via
vscode so there wasn't ...
0
votes
0
answers
68
views
FluentD unable to establish connection to ElasticSearch
I am trying to setup an FluentD + ECK on my Kubernetes Cluster.
But FluentD is failing to establish connection with ElasticSearch which is on SSL.
Error log
2022-10-12 04:55:27 +0000 [info]: adding ...
0
votes
0
answers
111
views
Regexp test in Fluentular passed but pattern not matched when use Fluentd to get logs
When I test regexp in Fluentular, response is in the form of json
This is my regexp:
/^((?<time>([0-9]{4})-(0[1-9]|1[0-2])-(0[1-9]|[1-2][0-9]|3[0-1]) [0-9]{1,2}:[0-9]{1,2}:[0-9]{1,2},[0-9]{3})\s)...
0
votes
1
answer
362
views
How to troubleshoot td-agent configuration file
I am learning to use FluentD. I am trying to use it to send logs to coralogix. I have installed the td-agent and I got it to work in a basic manner by using tail plugin to tail a logfile and output ...
0
votes
1
answer
81
views
Fluentd modify json output
I need to modify output to from this
"payload": {
"a": "value",
"b": "value",
.....
}
to
"a": "value",
"b": "...