Questions tagged [elastic-beats]
The elastic-beats tag has no usage guidance.
80
questions
0
votes
1
answer
21
views
Filebeat Cisco Module for Nexus having errors reading the config file
I am trying to set up syslogging from a nexus switch to feed into Filebeat's Cisco module that would then feed into Elasticsearch. I tend to get the same error message after enabling the cisco module ...
0
votes
0
answers
11
views
Capture network interface name with Packetbeat
With Packetbeat on Linux, the packetbeat.interfaces.device: any configuration captures all messages sent or received by the server where Packetbeat is installed.
I want to distinguish the messages ...
0
votes
0
answers
298
views
Elasticsearch filebeat failed to start
I have been struggling for quite some time with my filebeat setup.
I have installed filebeat 7.10 on an ubuntu instance. Somehow part of the logs were sent to my cluster, but now when I check the ...
1
vote
1
answer
189
views
How to monitor https traffic using packetbeat?
Is there way to get the response code of https calls using packetbeat. I don’t see any protocol for https. The goal is to monitor https traffic
0
votes
0
answers
89
views
How to monitor Publish and Delivery Rate of messages in RabbitMQ Queues using Metricbeat?
I am trying to monitor the publish and delivery rate of messages in RabbitMQ using Metricbeat, but I can't find any Metricbeat field that provides these metrics.
I have already configured Metricbeat ...
0
votes
1
answer
33
views
Observability section not visible in ELK Open Distro 1.13.3
I am using Open Distro 1.13.3 , Kibana v 7.10.2 but I am unable to see the observability section in Kibana. I am trying to implement HeartBeat and use uptime to monitor URL's but unable to see ...
0
votes
0
answers
29
views
AuditBeat: How to get resolved symbolic links in published event output?
I am using AuditBeat to monitor filesystem operations performed in an external application and a specific root path, e.g.: /var/myapp/myroot
Configuration
AuditBeat version: 7.14.1
auditd module ...
0
votes
1
answer
57
views
Recommended Docker Image Architecture
We have a "few" applications we want to run in a docker container. Our initial attempt created an "base image" that served as the basis for the different application images. It ...
0
votes
0
answers
98
views
Apache Nifi processor ListenBeats don't work correctly
I'm shiping windows events using ListenBeats in NiFi with winlogbeat, when the data arrived is duplicated in 2 FlowFiles. I don't know why.
This is the configuration about winlogbeat I'm using:
...
0
votes
1
answer
293
views
Is Elastic/Metricbeats suitable for process monitoring and alerting?
Do you use Elastic and Metricbeats for process monitoring and alerting? How did you configure your data gathering and alerting?
I am currently trying to set this up, and running into some basic issues....
0
votes
1
answer
178
views
Remove ECS data from metricbeat for smaller documents
I use the graphite beat to get graphite protocol metrics into es.
The metric document is much bigger than the metric data itself (timestamp, value, metric name).
I also get all the ECS data inserted ...
0
votes
1
answer
767
views
Filebeat is not forwarding nginx ingress controller logs
Filebeat 7.12.1
ECK operator 2.2
I'm trying to setup the filbeat for the Nginx-ingress access logs in my ECK stack (installed in GKE). I can access the logs directly in the pod but nothing is coming ...
0
votes
1
answer
327
views
decode_json_fields not moving data to root
We are using Winlogbeat to collect Event logs but rather than pull the data out of the winlog field, I want to move all the contents into the root field, which will help me automatically generate the ...
1
vote
1
answer
656
views
Running filebeat setup fails
I installed Elasticsearch and Kibana on docker. I need to add Filebeat to the workflow, so I followed the official documentation Run filebeat setup
I run this command:
docker run docker.elastic.co/...
0
votes
1
answer
133
views
Where does Beats send the data?
I have a theoretical question about Beats.
If I understand correctly, Logstach allows to easily parse the received data and ingest it to ElasticSearch. So my question is, does Beats send the retrieved ...
0
votes
1
answer
575
views
ElasticSearch Ingest Pipeline Null Value Question
If I am using an elasticsearch ingest pipeline in a Filebeat module, and I see the statement of:
if: 'ctx.json?.userIdentity?.userName == null'
When json.userIdentity.userName is equal to null, what ...
0
votes
1
answer
3k
views
If then else not working in FileBeat processor
I'm trying to setup some processors in a filebeat.yml to process some logs before sending to ELK.
An important part of the processing is determining the "level" of the event, which is not ...
4
votes
0
answers
471
views
Filebeat kafka input using multiline parser gives no output
Filebeat is configured to use input from kafka and output to file
When the multiline setting is turned off, the output is published to a file.
But when kafka input is configured with mutiline, no ...
1
vote
0
answers
486
views
Shipping logs from network share using Filebeat on Windows
The problem statement: I have an application running on Windows. I want to ship logs files from this application to ELK fronted by Kafka.
The challenge: This application writes a lot of process ...
0
votes
2
answers
568
views
Having 1 index vs multiple index in Elastic stack
Lets assume a condition if i have 500k logs in last 15mins which would be better for my elastic stack performance
Having 10 index to hold these 500k logs.
Having 1 index to hold 500k logs with more ...
0
votes
0
answers
122
views
Shipping a huge amount of logs from an unmanagable folder structure using filebeat
I am trying to ship log files to logstash using filebeat. The folder structure of the input path is currently very branchy and very big.
We have a following structure:
<LOG_PATH>/<STREAM>/&...
0
votes
0
answers
248
views
Can someone explain the detailed technical workflow for communication between elastic agent and fleet server
Need a deeper understanding of how authentication workflow is implemented during the first time installation of elastic agent and how the communication of elastic agent with the fleet server is ...
1
vote
3
answers
533
views
How to ingest a .json file from persistent volume to elasticSearch
I have a web crawler that crawls thousands of websites and store that in a persistent volume on Kubernetes.
After that pod is terminated, I want to ingest the .json file inside the PV into ECK. I have ...
1
vote
1
answer
495
views
ELK MetricBeat - Monitor remote mysqlDB
I have installed ELK stack (With metricbeat) on ServerA and want to monitor mysql on ServerB. I added db host details on ServerA mysql.yml metribeat module file (/etc/metricbeat/modules.d/mysql.yml)
...
1
vote
2
answers
4k
views
How to parse a mixed custom log using filebeat and processors
I'm trying to parse a custom log using only filebeat and processors. I wouldn't like to use Logstash and pipelines.
Below a sample of the log:
TID: [-1234] [] [2021-08-25 16:25:52,021] INFO {org.wso2....
0
votes
0
answers
720
views
memory management for elasticsearch
I am trying to calculate good balance of total memory in three node es cluster.
If I have three node e.s cluster each with 32G memory, 8 vcpu. Which combination would be more suitable for balancing ...
0
votes
0
answers
116
views
Adding Threat Intel fields in Elasticsearch data
How we can add threat intel data for elastic agent logs before they are entering into the Elasticsearch cluster?
I am trying to add some new fields to each log before based on the details present in ...
0
votes
0
answers
59
views
Possibility of configuring logstash to globally mutate index patterns from a Beats input stream
I have searched on the ELK discussion forum and received no answer. I had this task linked here "Logstash configuration to globally mutate sub index patterns" and it was correctly solved - ...
1
vote
0
answers
287
views
call jinja2 template within Ansible role at Playbook level
I have a folder structure as
Ansible:
- roles
- elastic.beats
- templates
filebeat-inputs.yml.j2
From my playbook, I am calling it as below:
- {
role: elastic.beats,
beat: "...
0
votes
1
answer
242
views
logstash - Do not get the values for [log][file][path]
I receive the data from filebeats in logstash but I do not see the value for [log][file][path].
I see othr people get the data in the logstash data, below is what I receive. There is no data from ...
0
votes
2
answers
176
views
Beats installation on AWS-ec2 to send to on-prem ELK
I have to setup jboss over AWS-EC2-Windows server, this will scale-up as well as per the requirements. We are using ELK for infrastructure monitoring for which will be installing beats here which will ...
0
votes
1
answer
1k
views
Multiple beats input ports allowed on Logstash configuration?
I'm trying to achieve getting Logstash to listen to 2 different beats ports - one for filebeat and one for winlogbeat, by setting the conf file to be -
input {
beats {
id => "...
1
vote
1
answer
734
views
Getting "could not be deployed. Enable it" while deploying Functionbeat as AWS Lambda with SQS trigger
I am trying to deploy Functionbeat to AWS Lambda from a Windows machine, I have followed the documentation I could find
What I did:
1- Downloaded Functionbeat from https://www.elastic.co/downloads/...
0
votes
1
answer
66
views
How to push complex legacy logs into logstash?
I'd like to use ELK to analyze and visualize our GxP Logs, created by our stoneold LIMS system.
At least the system runs on SLES but the whole logging structure is some kind of a mess.
I try to give ...
0
votes
3
answers
6k
views
Can't filebeat use multiple input/output config files in one instance?
Want to deploy filebeat with 3 log definations together. Send to different output targets.
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: filebeat
labels:
k8s-app: filebeat
spec:
...
0
votes
1
answer
1k
views
SSL Setup between Filebeat and Amazon MSK (Kafka)
I have successfully got Filebeat exporting logs to MSK in plaintext mode. When trying to do the same over SSL I’m getting error messages from the Filebeat side
Publish failed with circuit breaker is ...
0
votes
0
answers
591
views
Connect MySQL through JDBC in filebeats in Elastic Cloud
I am not able to connect MySQL in filebeats in Elastic Cloud.
No error I am getting.
Can anyone guide me how to do it ?
This configuration I'm using
input {
jdbc {
...
1
vote
1
answer
1k
views
Mock an ELK Beat output to Logstash with Postman
I have Elastic Beats (like metricbeat and auditbeat) that send data to logstash. For testing purposes, I'd like to mock some data from the beats to the logstash input.
I'm using the logstash output ...
5
votes
3
answers
19k
views
Logstash beats input "invalid version of beats protocol"
I'm writing a kibana plugin and a logstash pipeline. For my tests, I just wrote a logstash input like that:
input {
beats {
port => 9600
ssl => false
ssl_verify_mode =...
0
votes
0
answers
72
views
Is it possible to publish many events to ElasticSearch at once as a transaction? So if one publication fails, ES databse rolls back?
What I'm doing atm is: I gotta fetch a XML file from a server. This XML contains many logs and each log is parsed and turned into an event, a beat.event. What I want is, to send all logs to ...
0
votes
1
answer
393
views
ingesting nifi to hdfs to a single directory
Scenario
CSV data named test_csv.csv from windows.
Ingesting CSV data to hdfs. Beats > (ListenBeats) NiFi (PutHDFS) > HDFS
data sample:
a,b,c,d,e
a1,b1,c1,d1,e1
a2,b2,c2,d2,e2
a3,b3,c3,d3,e3
a4,...
3
votes
1
answer
524
views
Processing multiple logs from the same machine using ELK
I have a single host containing multiple log files. 2 such log files are csab and oneapplogs.
The corrseponding Groks are
opeapplog :
match => {"message" => "\[%{WORD:fixed}\|%{...
1
vote
0
answers
180
views
Filebeats saying dropping the line but still see in Kibana
I'm working on the ELK stack. I'm trying to push the wso2 application logs to Elastic Search.
I configured Filebeats to read the lines that include DCS.
I see the filebeats log in the terminal as ...
3
votes
1
answer
4k
views
Elastalert simplified multiple rules in one file
I'm writing Elastalart rules for heartbeat i.e if service or machine are/is down, I should get notified. Right now I can create one rule for service per one file like below.
name: My Alert
type: ...
1
vote
0
answers
47
views
How to handle overlapping logs with filebeat?
is there a way to handle overlapping logs with filebeat? here's a sample of what i have:
[17/02/2020 07:53:27:748] 00000 I >> message1 (start of event)
[17/02/2020 07:53:27:751] 00000 I @ ...
0
votes
1
answer
3k
views
Metricbeat WARN Cannot index event
I've seen a few posts like this, but none of them solved my problem, so:
I created a elastic cluster following this tutorial: https://www.elastic.co/guide/en/elasticsearch/reference/current/...
0
votes
1
answer
89
views
Running Beats with docker or without?
As Beats a Data shippers, and they suppose to ship files, metrics, and so on, from a machine, is it a good approach to run beats with docker, as in this way beats is containerized?
I have currently ...
0
votes
1
answer
264
views
Alternate for Filebeat
We are trying to implement ELK for my project and I'm very new to this stack.
Right now, we are facing a challenge to configure Filebeat in our servers due to old version of Linux.
Can anyone ...
2
votes
2
answers
1k
views
Problem with Kibana unable to load Dashboards (Exiting: resource 'metricbeat-7.5.0' exists, but it is not an alias)
I have installed metric-beat , but unable to load he kibana dashboard for mongodb.
i am getting below error,
[10.0.1.1:9300][indices:data/read/search[phase/query]]
Caused by: java.lang....
0
votes
1
answer
663
views
Can we access node/host process from the Kubernetes pod?
Is there a way to access the underlying host/node's process from Kubernetes pod in the same way like we access the host/node's filesystem by using hostPath volume mount?
PS: I am trying to monitor ...