Questions tagged [logstash]
Logstash is a tool for managing events and logs. You can use it to collect logs, parse them and send them to storage for later use (such as searching).
8,250
questions
0
votes
0
answers
15
views
logstash or filebeat to create multiple output files from tomcat log
I need to parse a tomcat log file and output it into several output files.
Each file is the result of a certain filter that will pick certain entries in the tomcat file that match a series of regexes ...
0
votes
0
answers
11
views
Logstash file chunk size
i have some files in a folder and i want to index them in Elasticsearch using logstash.
some of the files are indexed, but i get a lot of this error i don't know why :
[2022-06-30T16:15:24,966][INFO ][...
0
votes
1
answer
18
views
How to add data into nested field via Logstash output
I have a users index with a nested cars field.
How can I add the mysql result to the nested field of that user.
After some digging to the internet I found this code
POST /users/1/_update
{
"...
0
votes
1
answer
32
views
Logstash not ingesting content into elasticsearch
I have installed elasticsearch-8.2.3 logstash-8.2.3 and kibana-8.2.3 I have configure the logstash conf file to ingest content into elasticsearch, logstash run without any error but it is not ...
2
votes
0
answers
31
views
accessing the previous value in a field logstash
my document in elasticsearch looks like this:
"_index" : "micro_youtube",
"_type" : "video",
"_id" : "-i4trOrPnDE",
"_score" : 1.0,...
0
votes
1
answer
43
views
Aggregate exception occurred {:ERROR=>#<NoMethodERROR: undefined method `-’ for nil:NilClass>
I'm new with elastic, I'm working with log file , I want to process them so I can create a new fields based on aggregation to calculate time between operations "overall" and "internal&...
0
votes
1
answer
22
views
How to pass multiline environment variable in docker-compose.yml
I am trying to convert this docker run command:
docker run -it --rm --name logstash --net test opensearchproject/logstash-oss-with-opensearch-output-plugin:7.16.2 -e 'input { stdin { } } output {
...
0
votes
1
answer
28
views
How to filter out the file path using grok filter and add as new field
I want to filter out scm_output portion from /lib/logs/scm_output.log this file path and abc_output from this file path , /lib/logs/abc_output.log, and add new fields as scm_output & abc_output.
...
0
votes
0
answers
10
views
How to define logstashencoder in yml file
Could someone please help me to convert this logback-spring.xml to logback-spring.yml ?
<appender class="ch.qos.logback.core.ConsoleAppender" name="consoleAppender">
&...
-1
votes
0
answers
14
views
how dump/ingest nest API Json data to elastic search
I am looking for solution where I can get data from different nested REST API and dump them on elastic search on large scale, there are hundreds of APIs.
0
votes
0
answers
30
views
dotnet/tye and ELK docker on local machine - no indexes
I would like to setup my .NETCore microservice running via Dotnet/Tye project with local ELK docker.
I followed next guidelines:
https://github.com/dotnet/tye/blob/main/docs/recipes/logging_elastic.md ...
0
votes
0
answers
14
views
Getting Entity information from kibana after POST request
I'm trying to use JHipster V6.10.1 with MySQL and Jhipster-console to visualize Jhipster entities that are updated through POST requests. I'm able to see that requests went through from the Kibana ...
0
votes
0
answers
17
views
Create Optional param in grok pattern
Hi i am trying to create a Grok pattern for the following log event
2022-06-22 12:26:40,096 - INFO - test:Upgrade to websocket successful
In the above event, test is optional some events have it ...
0
votes
1
answer
28
views
Logstash Jdbc plugin filling more data in elasticsearch than the actual data, keeps on running
Logstash is running in an infinite loop and I'm having to stop the process, basically keeps filling values in the elasticsearch index. I need exact same number of documents as there are rows in my db ...
2
votes
0
answers
40
views
when i am creating multiple dashboard, it is streaming same logs table for all dashboard
I am using filebeat to ship my logs and using elasticsearch, logstash and kibana for visualization. On kibana dashboard, I have setup
Observability-> logs-> streams-> settings-> log index ...
0
votes
0
answers
16
views
How would I go about parsing log data in kibana from data that is already indexed?
So I work for a startup and we are using Aptible to host our elastic stack. I'm not sure if that affects anything but figured I'd mention it. I'm the only guy here with any working knowledge of the ...
-1
votes
1
answer
14
views
"\Logs was unexpected at this time" while starting logstash in my windows machine
I Don't know why this happensenter image description here
0
votes
1
answer
16
views
How to compare float in logstash?
It's the pipeline config for test:
input {
file {
path => "/tmp/test1.log"
}
}
filter {
json {
source => "message"
}
}
output {
if [a] == 1.1 {
stdout {...
0
votes
0
answers
29
views
Logstash 7.9.3 not starting
only this content shows up, while kibana and elastic search both are running
logstash.conf looks like this
0
votes
1
answer
24
views
Grok patterns logstach custom
hello im trying to get grok patern of this :
2589522 HARS-X6790 2022-08-05 08:55:15 --EX: SCR45778554 ID '346402454540629'
%{NUMBER:ligne} %{TIMESTAMP_ISO8601:syslog_timestamp} %{GREEDYDATA:...
0
votes
1
answer
35
views
How to push logs from kubernetes to elastic cloud deployment?
I am trying to configure logstash and filebeat running in kubernetes to connect and push logs from kubernetes cluster to my deployment in the elastic cloud.
I have configured the logstash.yaml file ...
-1
votes
0
answers
10
views
How to open and operate kibana after installation of elastic search and kibana on ubuntu?
I am very new to ELK stack, I have installed Logstash, elastic search and kibana. All the services are Up.
How to Open the kibana Dashboard in my machine now?
It look like below Image.
1
vote
1
answer
25
views
Logging stack trace in log appender
I would like to log the exception stack trace in stack-trace field of logstash log appender instead of being inside the message field
public void exceptionsCatcher(JoinPoint joinPoint, ...
0
votes
0
answers
17
views
Grok Pattern for dynamic jsons
My data is coming live from spring boot logs, In case of successful execution the logs show in JSON format (each JSON has different input fields) and for error, the system receives data in a string.
I ...
1
vote
1
answer
26
views
Logstash index based on filter
Hi I am reading stdout and filtering both apache and application logs using logstash as below
input {
beats {
port => 5044
}
}
filter {
grok {
match => { "message" => "%...
0
votes
1
answer
28
views
Logstash (ELK): Enrich IP with hostname (based off a file). - (No direct connect w/ DNS/AD)
Trying to figure out how to enrich the data being ingested (Network Data) Zeek/Suricata. I would like to either show Hostname vice IP, or more preferably add another field for hostname based off the ...
0
votes
0
answers
17
views
Is it possible to rename a kibana value for better graph readablity?
Given the following data:
records: [
{id: 1, path: "www.somewebsite.com/en/landing.html" },
{id: 2, path: "www.somewebsite.com/es/landing.html" },
]
is it possible to achieve ...
0
votes
1
answer
37
views
Logstash - Parsing Optional Logs through Custom grok pattern
Following is my log patterns
Sample 1 :
2022-06-14 02:03:22.051 INFO [ServiceName,TraceId,SpanID] 109171 --- [Thread] ClassName : A=ValueA B=ValueB C=ValueC
Sample 2:
2022-06-14 02:03:22.051 INFO ...
0
votes
1
answer
11
views
How to store Elasticsearch data in rdb?
hello? Due to a customer's request, we are in a situation where we need to input elasticsearch data into the rdb table.
Customers want this to happen in real time.
When inputting rdb data into ...
1
vote
1
answer
21
views
How can I do multiple input streams in logstash?
My use case is:
I need a timestamp from database A
Then do a query on database B
Then output the results of that Database B query
Which means a logstash file that (theoretically) would look ...
0
votes
2
answers
32
views
Logstash event field reference is not working
I have a small event that I want to pass to logstash from python:
{
"@timestamp":"2022-06-13T16:48:39.422Z",
"@version":"1",
"message":"...
0
votes
1
answer
19
views
How to match data in logstash one data input from API and second from csv file on server
I have tried all possible case but unable to match data between API and CSV file. my ultimate goal is read host key from REST API and matching with CSV file and print respective OS. Please suggest, ...
0
votes
0
answers
26
views
Logstash grok pattern for .txt files
I'm trying to grok below logline using the logstash filer
{"Type":"Page","Action":"Load","Time":"3/11/2012 9:32:17 PM","author":&...
0
votes
1
answer
29
views
Filebeat not reading logs from nested directories
I am relatively new to ELK stack and I am trying to send logs from a linux servers to elasticsearch. The path I am choosing is -
I have installed the filebeat on linux server where my application logs ...
0
votes
2
answers
18
views
Remove a middle-level field in Logstash
What I need
I'm using Logstash and trying to remove a field that is between the top-level field and the last field using the ruby filter. The top-level field name is always the same, only its ...
0
votes
1
answer
39
views
Elasticsearch output plugin in Logstash doesn't insert a document after deleting it previously
My app communicates with logstash, sending it objects. One of the options is upserting a document, and the other is deleting a document.
When doing one thing at the time it works perfect.
However, ...
1
vote
1
answer
76
views
Keycloak 17 logging to logstash
Does anyone have an idea on how I can do this (what the top comments suggests): Keycloak logging to logstash
But for the newest keycloak version 17+ which uses Quarkus instead of Wildfly?
0
votes
0
answers
38
views
Append value to an array in a single field through logstash in elasticsearch
I am using Logstash as the log collector for ingesting data in elasticsearch. I have one use case where I have to append a value (or array) to an existing array in a field.
Suppose, I have the array [...
0
votes
0
answers
15
views
Encoding issue when streaming logs from AWS Kinesis to ElasticSearch via Logstash
I've got an AWS Kinesis data stream called "otelpoc".
In Logstash, I'm using the Kinesis input plugin - see here.
My Logstash config is as follows:
input {
kinesis {
...
0
votes
1
answer
15
views
Have logstash JDBC input `last_run_metadata_path` on an external source?
We want to run logstash in an ECS environment where the instances have transient lives. They might be long lived, but they aren't permanent. Thus, having logstash store the sql_last_value in the ...
0
votes
0
answers
16
views
Logstash jdbc input does not retry as setup
I am using logstash to transfer data from postgresql to mysql.
There are 10 configure files in /etc/logstash/conf.d and I run logstash as a service by the command systemctl start logstash.
I have ...
0
votes
0
answers
27
views
How to take byte codes and combine into single line in logstash
I have config filebeat and logstash to enter one part of the log, but what I need here is the byte code that is in the log and put them together into a single line.
Here's the filebeat
filebeat.inputs:...
0
votes
0
answers
55
views
OpenSearch on AWS does not recognise GeoIP's location as GEOJSON type
I've got logstash processing logs and uploading to an opensearch instance running on AWS as a service.
I've added a geoip filter to my logstash to process IPs into geographic data. According to the ...
0
votes
1
answer
45
views
Kibana Not Showing Any Logs (ELK Stack)
I am learning ELK stack. So I have written a sample java code that churns out some log line in a file. I am trying to see if the log lines show up in Kibana UI. So I will try to explain it to best of ...
0
votes
0
answers
22
views
Logstash input twitter `Error::Forbidden`
I am running logstash 8.2.2 on Ubuntu 16,04 with the following command:
bin/logstash -f /etc/logstash/conf.d/twitter.conf
Here is the content of twitter.conf:
input {
twitter {
consumer_key => '...
1
vote
1
answer
25
views
GROK Pattern for Thread Name (LogStash 8.2.0)
I am new to LogStash. So I am learning GROK patterns and how to extract data from log lines. So I am using Logback to print log lines. The layout in the logback.xml is something like:
%date{dd/MM/YYYY ...
0
votes
0
answers
17
views
Filebeat not picking up some files
I am trying to send tomcat logs to ELK. I am using Filebeat to scan the files.
My log file name would be "project_err.DD-MM-YYYY". In filebeat configuration, I am giving the file name as ...
0
votes
0
answers
12
views
How to create the elasticsearch index document from s3 json file object
S3 to Elasticsearch
We have JSON files in s3, ship the s3 JSON object to elasticsearch as a document(one json files in s3 to one document elasticsearch)
Push s3 JSON objects (s3 containers json files),...
1
vote
0
answers
11
views
How to receive logs from multiple filebeats running on different machine(ec2) to one logstash instance?
i have 2 filebeats running on different machine(Ex: A & B) and filebeat is also configured to send logstash machine but logstash is not receiving the data even though I have added the private IP ...
0
votes
0
answers
59
views
Logstash container stopped because of an error creating action from filter
Hello I'm new to Elasticsearch
I'm working with log files comming from filebeat and logstash and I'm trying to add a field "response_time", and then affect the difference between timestamps ...