Questions tagged [logstash]
Logstash is a tool for managing events and logs. You can use it to collect logs, parse them and send them to storage for later use (such as searching).
8,607
questions
0
votes
0
answers
8
views
logstash sending to https server: [HTTP Output Failure] Could not fetch URL
client is using logstash to send to our https server, format is json. The sending rate is 500 eps, not high. We could get some data, but sometimes logstash side has below error. Could anyone help, ...
1
vote
1
answer
19
views
Logstash issue while following elastic documentation
As part of a training project, I am trying to setup a single instance of Logstash and test in standalone according to the elasticssearch documentation (https://www.elastic.co/guide/en/logstash/7.17/...
0
votes
0
answers
14
views
Logstash multiline codec doesn't work for stacktraces
I'm using the ELK stack to process my app's logs. So, the problem is that Logstash multiline codec doesn't work properly for the stacktraces. The input multiline logs are not parsed by the pattern and ...
-1
votes
0
answers
11
views
Multiple pipeline with batch size not getting batched in logstash
We are having 2 pipelines defined for logstash
- pipeline.id: pipe1
queue.type: persisted
path.queue: "/a/b/persistentQueue/"
path.config: "/a/*logstash.conf"
queue....
0
votes
1
answer
15
views
how to manually install logstash-jdbc-output plugin?
I cant have internet connection with host.
though I have the whole zip file of the plugin, from git.
so I have to manually install the plugin.
but sadly the git doc doesnt have manual install guide ...
0
votes
0
answers
10
views
Logstash: Nginx access logs agent field conflicts with ECS default agent field
I am trying to include agent field from Nginx access logs using the useragent
filter in Logstash to get client operating system data in ES, but the LS ingests the Filebeat related data to ES in the ...
1
vote
0
answers
17
views
What is the most common way to ingest JSON file changes from Azure BLOB storage into Logstash?
I have a need to integrate Logstash data ingestion to include JSON files which are stored in an Azure BLOB storage container. These JSON files are being constantly appended to and Logstash is expected ...
0
votes
0
answers
18
views
Logstash: how to install azureblob input plugin without running of heap memory
I'm running Logstash in Docker on Ubuntu 22 host with 16 GB of RAM.
This is my docker-compose.yml:
version: '3.7'
services:
tls:
profiles:
- setup
build:
context: tls/
...
0
votes
1
answer
13
views
Get data from elasticsearch request -gives multiple documents- from a logstash pipeline
I'm having a hard time trying to get some additional data from a elastic instance to a logstash pipeline. The elasticsearch filter doesnt give any exception errors, but no data is being populated in ...
0
votes
1
answer
10
views
Grok syntax and distinct if it is a new entry or continuation of previuoys one
I am using Grok as a default plugin for filtering my log, so let me say I have a simple 3 log entries:
2023-08-17 10:10:50.751 +02:00 [WARNING] [Provider] Failed to collect
2023-08-17 10:10:50.751 +02:...
0
votes
1
answer
14
views
Use logstash to analize log line
Hi I want to implement els stack for better log review, so I added logstash to ealsticsearch and kibana that already exists, logs are in form of files that was generated by .net apps that uses serilog ...
0
votes
0
answers
6
views
How can i ogstash input jdbc error handling
This is my
input {
jdbc {
jdbc_driver_library => "/usr/share/java/postgresql.jar"
jdbc_driver_class => "org.postgresql.Driver"
jdbc_connection_string => &...
0
votes
0
answers
19
views
How to process a JSON array with Logstash
I'm writing a Logstash pipeline to inject log files in JSON format.
Some of these input files contain records with a nested JSON arrays.
This pipeline needs to index each of the JSON objects within ...
0
votes
1
answer
15
views
Logstash is unable to parse a valid JSON file
I have the following JSON file:
{ "UserId":"433157", "Username":"Thu Huynh", "Email":"THuynh+1@sphera.com", "Timezone":"...
0
votes
0
answers
23
views
Using a free database for the Geoip logstash feature
I have enabled the geoip of logstash plugin, but I cannot manually insert the database I want into the config file and it keeps giving the following error.
I didn't understand this exactly, the geoip ...
0
votes
0
answers
21
views
LogStash in a loop
I am running Elasticsearch, Kibana and Logstash version 8.9.1.
My Logstash is in a loop:
[2023-09-08T22:05:38,932][DEBUG][org.logstash.instrument.metrics.ExtendedFlowMetric] RetentionWindow{policy=...
0
votes
0
answers
18
views
How to extract the specific field from JSON log saved in field message using Logback and Logstash?
I'm using Zalando Logbook to generate JSON log messages in my Java application. The log messages are configured with a LoggingEventCompositeJsonEncoder in Logback. Now, I want to extract the "...
0
votes
0
answers
30
views
Issues while Trying to setup elk version 8.9.1 using docker
I'm trying to setup elk stack in docker and I'm facing few issues.
by using docker compose - I'm not able to login into kibana dashboard, as I'm not able to get the passcode.
I tried to setup ...
0
votes
0
answers
17
views
Logstash in k8s - parsing nested json from MongoDB and get every nested json as separated field
I'm using Logstash to take documents from specific MongoDB collection, and save it to Elasticsearch.
Nested fields are being saved to "log_entry" as one JSON, starting with "BSON" ...
0
votes
1
answer
48
views
Logstash does not ingest GeoIP data to Elastic Search
I am new to ELK and I have deployed Elastic and Kibana on K8s, I would like to monitor an standalone Nginx server which is running in a standalone Ubuntu server, I have installed the filebeat and ...
-1
votes
0
answers
20
views
What is the Log pattern to log caller details with Flogger in spring boot
I am using Flogger API to log details in my spring boot controller.
I need to log caller details like class name, method name and line number. so, I have been using this log pattern:
Class:%class{36} ...
0
votes
1
answer
26
views
How to send logs to logstash from nestjs application using nestjs-pino?
I have the following configuration code in my AppModule in nestjs:
LoggerModule.forRoot({
pinoHttp: {
base: undefined,
autoLogging: false,
genReqId: () => uuidv4(),
...
0
votes
0
answers
13
views
how to map multiple filebeat inputs to logstash configuration in official helm chart?
I used following pipline in my installioan of logstash on centos machines.
how to map the following pipleline to official helm chart for logstash?
-pipeline.id: beats-server
config.string: |
...
0
votes
1
answer
51
views
rsyslog mmextenal logstash json output with escaped quotations and additional quotations
I'm having an issue with getting rsyslog to generate the correct format for logstash. I'm receiving remote rsyslog messages (using RELP) and I'm then processing those messages using some bespoke code ...
0
votes
1
answer
83
views
Sending logs of (rootless) podman containers to ELK?
I am struggling with the $question for a while and it is time to ask for some guidance.
We have 10+ containers that runs on different RHEL VMs deployed via ansible as systemd service (in other words, ...
1
vote
0
answers
99
views
Display and aggregate data with Logstash and Kibana
I am currently trying to get log data from my Jenkinses and display dashboards in Kibana.
Infrastructure:
I have multiple Jenkins instances running multiple different jobs.
With Filebeat, each log ...
0
votes
0
answers
26
views
Exception in Docker Compose: Issues While Running My Docker Compose Configuration
I'm currently facing a perplexing issue with my Docker Compose setup involving Logstash. I've been following various tutorials to configure Logstash within an ELK (Elasticsearch, Logstash, Kibana) ...
0
votes
0
answers
17
views
Data parse from multiple rsyslog to logstash to elasticsearch
I am new to ELK. I have deployed my ELK with this article.
https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elastic-stack-on-ubuntu-22-04
I skipped the ...
0
votes
0
answers
10
views
Logs Pulling Architecture
Filebeat uses a "push" architecture, sending logs to logstash or elastic. Is there a feature or product that would enable elastic to receive hosts' logs using a "pull" architecture?...
0
votes
0
answers
34
views
How to stream from Oracle Cloud Streaming to Elasticsearch Kafka Input Plugin?
I have set up Oracle Cloud Streaming all the way from pools to the stream itself configured as public, I am trying to use Logstash to pull the data and get it stored in Elasticsearch, and want to know ...
0
votes
1
answer
69
views
Unable to configure plugins: (ArgumentError) Cannot determine timezone from nil\n(secs
[2023-08-21T20:43:58,795][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::...
0
votes
0
answers
15
views
Docker logstash trying to connect to despite
I have a docker logstash configured as such:
input {
tcp {
port => 5000
}
}
output {
elasticsearch {
hosts => ["https://my-custom-elastic.com:9321"]
Expected:
I would ...
0
votes
0
answers
27
views
error 'using JAVA_HOME while Logstash distribution comes with a bundled JDK \IBM was unexpected at this time.' when trying to start logstash on my pc
I'm trying to start logstash on my machine, but got below error
C:\Users\xxx\Tool\logstash-7.13.3\bin>logstash -f ../test_cyrark_proxy.conf
"test"
Using JAVA_HOME defined java: "C:\...
0
votes
1
answer
47
views
Logstash config doesn't parse JSON file and not push data to index - log: {:count=>1,:running_pipelines=>[:main], :non_running_pipelines=>[]}
I generate a JSON file every 5 mins through a Python code and try to pushing data to Elastic, but the Logstash throw following message and doesn't push any data to Kibana.
My pipeline: File --> ...
1
vote
1
answer
43
views
Filebeat Processor isn't parsing field exception from json log file
I want to ingest JSON log files from an application into ElasticSearch. To do so, I am using Custom log files integration running on an Elastic agent, that is managed by a Fleet Manager.
The log files ...
0
votes
0
answers
71
views
Elasticsearch Logstash Kibana (ELK stack) + filebeat setup on ubuntu 22.04 AWS EC2 instance on t3.large to visualize system logs
I am trying to setup ELK stack + filebeat on ubuntu 22.04 AWS EC2 instance on t3.large to visualize system logs.
below is the process i am following
(Note - the steps are not copied from ChatGPT, the ...
0
votes
0
answers
19
views
Logstash docker image - No logstash-plugin?
I have deployed LogStash on kubernetes using the latest image (logstash:8.8.1) and now I would like to install some plugins.
I understand that I would need to use the 'logstash-plugin' tool, whis ...
0
votes
0
answers
14
views
can Logstash fetch data from 2 different sources in single Input JDBC section?
I am working on migrating logstash pipelines to read data from snowflake instead of sql. While most of my source data is available in snowflake, Couple tables are meant to be still in sql database as ...
0
votes
0
answers
17
views
add field in logstash with a stripped information from another field
Is there a valid function in logstash "add field" to strip a hostname to a part of it.
i have variations of hostnames from a haproxy.log like this
cccrz1-qa01.exserver.int
cccrz2-qa01....
0
votes
0
answers
11
views
Pausing Logstash During Synonym Updates
I'm currently engaged in a project that utilizes Elastic Search. Within this project, I've established a logstash pipeline that operates at 10-second intervals to ensure the synchronization of data ...
0
votes
0
answers
49
views
Masking nested objects in Spring boot using logstash-logback-encoder
I want to mask the password field of the log-
{"timestamp":"2021-10-26T22:41:29.471+02:00","level":"INFO", "body":[{"id":1,"password&...
1
vote
0
answers
51
views
Getting error : LOGBACK: No context given for net.logstash.logback.LogstashFormatter
I am trying to print a log in the SpringBoot Application using Logback and the following configuration, but the logs are not printing.
Spring-boot version 2.7.2 and Java version 17 are being used.
In ...
0
votes
1
answer
46
views
Does ECK required logstash
I'm quite confused between ECK and ELK. AFIK ELK requires logstash for reading the logs and whether the logstash required for ECK also or only filebeat or metric beat is enough to get logs. Filebeat ...
0
votes
0
answers
32
views
Grok Pattern and Hostname Extraction Issues in Logstash 8.8.2
List item
I am encountering an issue with Logstash 8 when processing logs from a client machine, while Logstash 6 correctly processes the same logs. Here are the log samples:
Log Sample - Logstash 8 (...
0
votes
0
answers
11
views
Secure Logstash and Filebeats communication
We are working on an integration where we need to take logs from Filebeat through Logstash. However, Filebeat and Logstash are hosted in different networks. In order to secure the communication, we ...
0
votes
1
answer
34
views
Logstash how to send data from a file in a loop?
As part of load testing, I have a couple of premade log files that simulate different scenarios.
I'm trying to use Logstash to send the content of 1 file, and resend it again in a loop every time it ...
0
votes
1
answer
39
views
ELK with Filebeat without Log File
I have setup my ELK stack with Filebeat to ship my logs, all works fine if I have a log file and add that path to my filebeat.yml
But what if my application has no log files generated and only using ...
0
votes
1
answer
43
views
Logstash http filter with array fields in body
I want to create an HTTP Logstash filter that invokes an API using a request body that includes an array field. The input plugin reads a document as follows:
{
"tags": ["foo", &...
0
votes
0
answers
77
views
LogStash cannot connect Postgresql and ElasticSearch in docker-compose
I'm trying to run Elastic Stack on docker-compose and import data from postgresql, but LogStash cannot connect to postgresql and ElasticSearch.
docker-compose.yml:
version: '3'
services:
db:
...
0
votes
0
answers
15
views
Logstash- How to parse formatted JSON arrays in log files
Logstash version: 7.17.10
Elasticsearch version:7.17.10
The logs are located in /var/logs directory and the format is as below:
[
{
"t": "SYS",
"dt": "2023-...