Questions tagged [elastic-beats]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
1 answer
21 views

Filebeat Cisco Module for Nexus having errors reading the config file

I am trying to set up syslogging from a nexus switch to feed into Filebeat's Cisco module that would then feed into Elasticsearch. I tend to get the same error message after enabling the cisco module ...
Legend27's user avatar
0 votes
0 answers
11 views

Capture network interface name with Packetbeat

With Packetbeat on Linux, the packetbeat.interfaces.device: any configuration captures all messages sent or received by the server where Packetbeat is installed. I want to distinguish the messages ...
Tabish Mir's user avatar
0 votes
0 answers
298 views

Elasticsearch filebeat failed to start

I have been struggling for quite some time with my filebeat setup. I have installed filebeat 7.10 on an ubuntu instance. Somehow part of the logs were sent to my cluster, but now when I check the ...
Terchila Marian's user avatar
1 vote
1 answer
189 views

How to monitor https traffic using packetbeat?

Is there way to get the response code of https calls using packetbeat. I don’t see any protocol for https. The goal is to monitor https traffic
Sowmiya 's user avatar
0 votes
0 answers
89 views

How to monitor Publish and Delivery Rate of messages in RabbitMQ Queues using Metricbeat?

I am trying to monitor the publish and delivery rate of messages in RabbitMQ using Metricbeat, but I can't find any Metricbeat field that provides these metrics. I have already configured Metricbeat ...
Karan Vyas's user avatar
0 votes
1 answer
33 views

Observability section not visible in ELK Open Distro 1.13.3

I am using Open Distro 1.13.3 , Kibana v 7.10.2 but I am unable to see the observability section in Kibana. I am trying to implement HeartBeat and use uptime to monitor URL's but unable to see ...
Navin's user avatar
  • 684
0 votes
0 answers
29 views

AuditBeat: How to get resolved symbolic links in published event output?

I am using AuditBeat to monitor filesystem operations performed in an external application and a specific root path, e.g.: /var/myapp/myroot Configuration AuditBeat version: 7.14.1 auditd module ...
Henrik Steudel's user avatar
0 votes
1 answer
57 views

Recommended Docker Image Architecture

We have a "few" applications we want to run in a docker container. Our initial attempt created an "base image" that served as the basis for the different application images. It ...
Mike Rother's user avatar
0 votes
0 answers
98 views

Apache Nifi processor ListenBeats don't work correctly

I'm shiping windows events using ListenBeats in NiFi with winlogbeat, when the data arrived is duplicated in 2 FlowFiles. I don't know why. This is the configuration about winlogbeat I'm using: ...
IamYipi's user avatar
  • 13
0 votes
1 answer
293 views

Is Elastic/Metricbeats suitable for process monitoring and alerting?

Do you use Elastic and Metricbeats for process monitoring and alerting? How did you configure your data gathering and alerting? I am currently trying to set this up, and running into some basic issues....
EdwardTeach's user avatar
0 votes
1 answer
178 views

Remove ECS data from metricbeat for smaller documents

I use the graphite beat to get graphite protocol metrics into es. The metric document is much bigger than the metric data itself (timestamp, value, metric name). I also get all the ECS data inserted ...
idan ahal's user avatar
  • 767
0 votes
1 answer
767 views

Filebeat is not forwarding nginx ingress controller logs

Filebeat 7.12.1 ECK operator 2.2 I'm trying to setup the filbeat for the Nginx-ingress access logs in my ECK stack (installed in GKE). I can access the logs directly in the pod but nothing is coming ...
Sameer Malhotra's user avatar
0 votes
1 answer
327 views

decode_json_fields not moving data to root

We are using Winlogbeat to collect Event logs but rather than pull the data out of the winlog field, I want to move all the contents into the root field, which will help me automatically generate the ...
IsaacK's user avatar
  • 1
1 vote
1 answer
656 views

Running filebeat setup fails

I installed Elasticsearch and Kibana on docker. I need to add Filebeat to the workflow, so I followed the official documentation Run filebeat setup I run this command: docker run docker.elastic.co/...
Abdelmoughit Gardam's user avatar
0 votes
1 answer
133 views

Where does Beats send the data?

I have a theoretical question about Beats. If I understand correctly, Logstach allows to easily parse the received data and ingest it to ElasticSearch. So my question is, does Beats send the retrieved ...
Sénéchal Julien's user avatar
0 votes
1 answer
575 views

ElasticSearch Ingest Pipeline Null Value Question

If I am using an elasticsearch ingest pipeline in a Filebeat module, and I see the statement of: if: 'ctx.json?.userIdentity?.userName == null' When json.userIdentity.userName is equal to null, what ...
Mike R's user avatar
  • 464
0 votes
1 answer
3k views

If then else not working in FileBeat processor

I'm trying to setup some processors in a filebeat.yml to process some logs before sending to ELK. An important part of the processing is determining the "level" of the event, which is not ...
ziv's user avatar
  • 27
4 votes
0 answers
471 views

Filebeat kafka input using multiline parser gives no output

Filebeat is configured to use input from kafka and output to file When the multiline setting is turned off, the output is published to a file. But when kafka input is configured with mutiline, no ...
Sniper's user avatar
  • 1,428
1 vote
0 answers
486 views

Shipping logs from network share using Filebeat on Windows

The problem statement: I have an application running on Windows. I want to ship logs files from this application to ELK fronted by Kafka. The challenge: This application writes a lot of process ...
Prabodh Mhalgi's user avatar
0 votes
2 answers
568 views

Having 1 index vs multiple index in Elastic stack

Lets assume a condition if i have 500k logs in last 15mins which would be better for my elastic stack performance Having 10 index to hold these 500k logs. Having 1 index to hold 500k logs with more ...
user avatar
0 votes
0 answers
122 views

Shipping a huge amount of logs from an unmanagable folder structure using filebeat

I am trying to ship log files to logstash using filebeat. The folder structure of the input path is currently very branchy and very big. We have a following structure: <LOG_PATH>/<STREAM>/&...
Luka Klarić's user avatar
0 votes
0 answers
248 views

Can someone explain the detailed technical workflow for communication between elastic agent and fleet server

Need a deeper understanding of how authentication workflow is implemented during the first time installation of elastic agent and how the communication of elastic agent with the fleet server is ...
Mann Mehta's user avatar
1 vote
3 answers
533 views

How to ingest a .json file from persistent volume to elasticSearch

I have a web crawler that crawls thousands of websites and store that in a persistent volume on Kubernetes. After that pod is terminated, I want to ingest the .json file inside the PV into ECK. I have ...
Jad Sakr's user avatar
1 vote
1 answer
495 views

ELK MetricBeat - Monitor remote mysqlDB

I have installed ELK stack (With metricbeat) on ServerA and want to monitor mysql on ServerB. I added db host details on ServerA mysql.yml metribeat module file (/etc/metricbeat/modules.d/mysql.yml) ...
ashwin's user avatar
  • 332
1 vote
2 answers
4k views

How to parse a mixed custom log using filebeat and processors

I'm trying to parse a custom log using only filebeat and processors. I wouldn't like to use Logstash and pipelines. Below a sample of the log: TID: [-1234] [] [2021-08-25 16:25:52,021] INFO {org.wso2....
Kleyson Rios's user avatar
  • 2,557
0 votes
0 answers
720 views

memory management for elasticsearch

I am trying to calculate good balance of total memory in three node es cluster. If I have three node e.s cluster each with 32G memory, 8 vcpu. Which combination would be more suitable for balancing ...
prat's user avatar
  • 103
0 votes
0 answers
116 views

Adding Threat Intel fields in Elasticsearch data

How we can add threat intel data for elastic agent logs before they are entering into the Elasticsearch cluster? I am trying to add some new fields to each log before based on the details present in ...
Pradeep M Nayak's user avatar
0 votes
0 answers
59 views

Possibility of configuring logstash to globally mutate index patterns from a Beats input stream

I have searched on the ELK discussion forum and received no answer. I had this task linked here "Logstash configuration to globally mutate sub index patterns" and it was correctly solved - ...
es-g's user avatar
  • 3
1 vote
0 answers
287 views

call jinja2 template within Ansible role at Playbook level

I have a folder structure as Ansible: - roles - elastic.beats - templates filebeat-inputs.yml.j2 From my playbook, I am calling it as below: - { role: elastic.beats, beat: "...
Buggy B's user avatar
  • 643
0 votes
1 answer
242 views

logstash - Do not get the values for [log][file][path]

I receive the data from filebeats in logstash but I do not see the value for [log][file][path]. I see othr people get the data in the logstash data, below is what I receive. There is no data from ...
BKruger's user avatar
  • 31
0 votes
2 answers
176 views

Beats installation on AWS-ec2 to send to on-prem ELK

I have to setup jboss over AWS-EC2-Windows server, this will scale-up as well as per the requirements. We are using ELK for infrastructure monitoring for which will be installing beats here which will ...
Abhishek's user avatar
0 votes
1 answer
1k views

Multiple beats input ports allowed on Logstash configuration?

I'm trying to achieve getting Logstash to listen to 2 different beats ports - one for filebeat and one for winlogbeat, by setting the conf file to be - input { beats { id => "...
omkar.ghaisas's user avatar
1 vote
1 answer
734 views

Getting "could not be deployed. Enable it" while deploying Functionbeat as AWS Lambda with SQS trigger

I am trying to deploy Functionbeat to AWS Lambda from a Windows machine, I have followed the documentation I could find What I did: 1- Downloaded Functionbeat from https://www.elastic.co/downloads/...
Yahya Hussein's user avatar
0 votes
1 answer
66 views

How to push complex legacy logs into logstash?

I'd like to use ELK to analyze and visualize our GxP Logs, created by our stoneold LIMS system. At least the system runs on SLES but the whole logging structure is some kind of a mess. I try to give ...
Alexander's user avatar
0 votes
3 answers
6k views

Can't filebeat use multiple input/output config files in one instance?

Want to deploy filebeat with 3 log definations together. Send to different output targets. --- apiVersion: apps/v1 kind: DaemonSet metadata: name: filebeat labels: k8s-app: filebeat spec: ...
iooi's user avatar
  • 453
0 votes
1 answer
1k views

SSL Setup between Filebeat and Amazon MSK (Kafka)

I have successfully got Filebeat exporting logs to MSK in plaintext mode. When trying to do the same over SSL I’m getting error messages from the Filebeat side Publish failed with circuit breaker is ...
Garreth's user avatar
  • 1,057
0 votes
0 answers
591 views

Connect MySQL through JDBC in filebeats in Elastic Cloud

I am not able to connect MySQL in filebeats in Elastic Cloud. No error I am getting. Can anyone guide me how to do it ? This configuration I'm using input {     jdbc {         ...
Tech Geek's user avatar
  • 425
1 vote
1 answer
1k views

Mock an ELK Beat output to Logstash with Postman

I have Elastic Beats (like metricbeat and auditbeat) that send data to logstash. For testing purposes, I'd like to mock some data from the beats to the logstash input. I'm using the logstash output ...
Mathieu Rollet's user avatar
5 votes
3 answers
19k views

Logstash beats input "invalid version of beats protocol"

I'm writing a kibana plugin and a logstash pipeline. For my tests, I just wrote a logstash input like that: input { beats { port => 9600 ssl => false ssl_verify_mode =...
Pikappa's user avatar
  • 191
0 votes
0 answers
72 views

Is it possible to publish many events to ElasticSearch at once as a transaction? So if one publication fails, ES databse rolls back?

What I'm doing atm is: I gotta fetch a XML file from a server. This XML contains many logs and each log is parsed and turned into an event, a beat.event. What I want is, to send all logs to ...
Pessoa's user avatar
  • 1
0 votes
1 answer
393 views

ingesting nifi to hdfs to a single directory

Scenario CSV data named test_csv.csv from windows. Ingesting CSV data to hdfs. Beats > (ListenBeats) NiFi (PutHDFS) > HDFS data sample: a,b,c,d,e a1,b1,c1,d1,e1 a2,b2,c2,d2,e2 a3,b3,c3,d3,e3 a4,...
yuliansen's user avatar
  • 470
3 votes
1 answer
524 views

Processing multiple logs from the same machine using ELK

I have a single host containing multiple log files. 2 such log files are csab and oneapplogs. The corrseponding Groks are opeapplog : match => {"message" => "\[%{WORD:fixed}\|%{...
Tony Stark's user avatar
1 vote
0 answers
180 views

Filebeats saying dropping the line but still see in Kibana

I'm working on the ELK stack. I'm trying to push the wso2 application logs to Elastic Search. I configured Filebeats to read the lines that include DCS. I see the filebeats log in the terminal as ...
saitama's user avatar
  • 719
3 votes
1 answer
4k views

Elastalert simplified multiple rules in one file

I'm writing Elastalart rules for heartbeat i.e if service or machine are/is down, I should get notified. Right now I can create one rule for service per one file like below. name: My Alert type: ...
Veerendra K's user avatar
  • 2,145
1 vote
0 answers
47 views

How to handle overlapping logs with filebeat?

is there a way to handle overlapping logs with filebeat? here's a sample of what i have: [17/02/2020 07:53:27:748] 00000 I >> message1 (start of event) [17/02/2020 07:53:27:751] 00000 I @ ...
Amine Maalfi's user avatar
0 votes
1 answer
3k views

Metricbeat WARN Cannot index event

I've seen a few posts like this, but none of them solved my problem, so: I created a elastic cluster following this tutorial: https://www.elastic.co/guide/en/elasticsearch/reference/current/...
Lucas Abbade's user avatar
0 votes
1 answer
89 views

Running Beats with docker or without?

As Beats a Data shippers, and they suppose to ship files, metrics, and so on, from a machine, is it a good approach to run beats with docker, as in this way beats is containerized? I have currently ...
Luka Klarić's user avatar
0 votes
1 answer
264 views

Alternate for Filebeat

We are trying to implement ELK for my project and I'm very new to this stack. Right now, we are facing a challenge to configure Filebeat in our servers due to old version of Linux. Can anyone ...
Kapila's user avatar
  • 85
2 votes
2 answers
1k views

Problem with Kibana unable to load Dashboards (Exiting: resource 'metricbeat-7.5.0' exists, but it is not an alias)

I have installed metric-beat , but unable to load he kibana dashboard for mongodb. i am getting below error, [10.0.1.1:9300][indices:data/read/search[phase/query]] Caused by: java.lang....
Chandu's user avatar
  • 21
0 votes
1 answer
663 views

Can we access node/host process from the Kubernetes pod?

Is there a way to access the underlying host/node's process from Kubernetes pod in the same way like we access the host/node's filesystem by using hostPath volume mount? PS: I am trying to monitor ...
Rajat Badjatya's user avatar