Questions tagged [logstash]

Logstash is a tool for managing events and logs. You can use it to collect logs, parse them and send them to storage for later use (such as searching).

Filter by
Sorted by
Tagged with
0 votes
0 answers
8 views

logstash sending to https server: [HTTP Output Failure] Could not fetch URL

client is using logstash to send to our https server, format is json. The sending rate is 500 eps, not high. We could get some data, but sometimes logstash side has below error. Could anyone help, ...
nathan's user avatar
  • 754
1 vote
1 answer
19 views

Logstash issue while following elastic documentation

As part of a training project, I am trying to setup a single instance of Logstash and test in standalone according to the elasticssearch documentation (https://www.elastic.co/guide/en/logstash/7.17/...
VenomCA's user avatar
  • 106
0 votes
0 answers
14 views

Logstash multiline codec doesn't work for stacktraces

I'm using the ELK stack to process my app's logs. So, the problem is that Logstash multiline codec doesn't work properly for the stacktraces. The input multiline logs are not parsed by the pattern and ...
Anna Karpunicheva's user avatar
-1 votes
0 answers
11 views

Multiple pipeline with batch size not getting batched in logstash

We are having 2 pipelines defined for logstash - pipeline.id: pipe1 queue.type: persisted path.queue: "/a/b/persistentQueue/" path.config: "/a/*logstash.conf" queue....
jay's user avatar
  • 1
0 votes
1 answer
15 views

how to manually install logstash-jdbc-output plugin?

I cant have internet connection with host. though I have the whole zip file of the plugin, from git. so I have to manually install the plugin. but sadly the git doc doesnt have manual install guide ...
muchdumber's user avatar
0 votes
0 answers
10 views

Logstash: Nginx access logs agent field conflicts with ECS default agent field

I am trying to include agent field from Nginx access logs using the useragent filter in Logstash to get client operating system data in ES, but the LS ingests the Filebeat related data to ES in the ...
Noor Muradi's user avatar
1 vote
0 answers
17 views

What is the most common way to ingest JSON file changes from Azure BLOB storage into Logstash?

I have a need to integrate Logstash data ingestion to include JSON files which are stored in an Azure BLOB storage container. These JSON files are being constantly appended to and Logstash is expected ...
Eugene Goldberg's user avatar
0 votes
0 answers
18 views

Logstash: how to install azureblob input plugin without running of heap memory

I'm running Logstash in Docker on Ubuntu 22 host with 16 GB of RAM. This is my docker-compose.yml: version: '3.7' services: tls: profiles: - setup build: context: tls/ ...
Eugene Goldberg's user avatar
0 votes
1 answer
13 views

Get data from elasticsearch request -gives multiple documents- from a logstash pipeline

I'm having a hard time trying to get some additional data from a elastic instance to a logstash pipeline. The elasticsearch filter doesnt give any exception errors, but no data is being populated in ...
gl4ssiest's user avatar
  • 167
0 votes
1 answer
10 views

Grok syntax and distinct if it is a new entry or continuation of previuoys one

I am using Grok as a default plugin for filtering my log, so let me say I have a simple 3 log entries: 2023-08-17 10:10:50.751 +02:00 [WARNING] [Provider] Failed to collect 2023-08-17 10:10:50.751 +02:...
Wojciech Szabowicz's user avatar
0 votes
1 answer
14 views

Use logstash to analize log line

Hi I want to implement els stack for better log review, so I added logstash to ealsticsearch and kibana that already exists, logs are in form of files that was generated by .net apps that uses serilog ...
Wojciech Szabowicz's user avatar
0 votes
0 answers
6 views

How can i ogstash input jdbc error handling

This is my input { jdbc { jdbc_driver_library => "/usr/share/java/postgresql.jar" jdbc_driver_class => "org.postgresql.Driver" jdbc_connection_string => &...
hi-inbeom's user avatar
0 votes
0 answers
19 views

How to process a JSON array with Logstash

I'm writing a Logstash pipeline to inject log files in JSON format. Some of these input files contain records with a nested JSON arrays. This pipeline needs to index each of the JSON objects within ...
Eugene Goldberg's user avatar
0 votes
1 answer
15 views

Logstash is unable to parse a valid JSON file

I have the following JSON file: { "UserId":"433157", "Username":"Thu Huynh", "Email":"THuynh+1@sphera.com", "Timezone":"...
Eugene Goldberg's user avatar
0 votes
0 answers
23 views

Using a free database for the Geoip logstash feature

I have enabled the geoip of logstash plugin, but I cannot manually insert the database I want into the config file and it keeps giving the following error. I didn't understand this exactly, the geoip ...
Amirhosseinramezan's user avatar
0 votes
0 answers
21 views

LogStash in a loop

I am running Elasticsearch, Kibana and Logstash version 8.9.1. My Logstash is in a loop: [2023-09-08T22:05:38,932][DEBUG][org.logstash.instrument.metrics.ExtendedFlowMetric] RetentionWindow{policy=...
aws-cloud-dfir's user avatar
0 votes
0 answers
18 views

How to extract the specific field from JSON log saved in field message using Logback and Logstash?

I'm using Zalando Logbook to generate JSON log messages in my Java application. The log messages are configured with a LoggingEventCompositeJsonEncoder in Logback. Now, I want to extract the "...
Piotr's user avatar
  • 569
0 votes
0 answers
30 views

Issues while Trying to setup elk version 8.9.1 using docker

I'm trying to setup elk stack in docker and I'm facing few issues. by using docker compose - I'm not able to login into kibana dashboard, as I'm not able to get the passcode. I tried to setup ...
onebyi3's user avatar
0 votes
0 answers
17 views

Logstash in k8s - parsing nested json from MongoDB and get every nested json as separated field

I'm using Logstash to take documents from specific MongoDB collection, and save it to Elasticsearch. Nested fields are being saved to "log_entry" as one JSON, starting with "BSON" ...
Denis's user avatar
  • 499
0 votes
1 answer
48 views

Logstash does not ingest GeoIP data to Elastic Search

I am new to ELK and I have deployed Elastic and Kibana on K8s, I would like to monitor an standalone Nginx server which is running in a standalone Ubuntu server, I have installed the filebeat and ...
Noor Muradi's user avatar
-1 votes
0 answers
20 views

What is the Log pattern to log caller details with Flogger in spring boot

I am using Flogger API to log details in my spring boot controller. I need to log caller details like class name, method name and line number. so, I have been using this log pattern: Class:%class{36} ...
Charu Jain's user avatar
0 votes
1 answer
26 views

How to send logs to logstash from nestjs application using nestjs-pino?

I have the following configuration code in my AppModule in nestjs: LoggerModule.forRoot({ pinoHttp: { base: undefined, autoLogging: false, genReqId: () => uuidv4(), ...
Sardor Taylakov's user avatar
0 votes
0 answers
13 views

how to map multiple filebeat inputs to logstash configuration in official helm chart?

I used following pipline in my installioan of logstash on centos machines. how to map the following pipleline to official helm chart for logstash? -pipeline.id: beats-server config.string: | ...
Ali farahzadi's user avatar
0 votes
1 answer
51 views

rsyslog mmextenal logstash json output with escaped quotations and additional quotations

I'm having an issue with getting rsyslog to generate the correct format for logstash. I'm receiving remote rsyslog messages (using RELP) and I'm then processing those messages using some bespoke code ...
Sean Lennon's user avatar
0 votes
1 answer
83 views

Sending logs of (rootless) podman containers to ELK?

I am struggling with the $question for a while and it is time to ask for some guidance. We have 10+ containers that runs on different RHEL VMs deployed via ansible as systemd service (in other words, ...
m4gic's user avatar
  • 1,471
1 vote
0 answers
99 views

Display and aggregate data with Logstash and Kibana

I am currently trying to get log data from my Jenkinses and display dashboards in Kibana. Infrastructure: I have multiple Jenkins instances running multiple different jobs. With Filebeat, each log ...
Nico's user avatar
  • 154
0 votes
0 answers
26 views

Exception in Docker Compose: Issues While Running My Docker Compose Configuration

I'm currently facing a perplexing issue with my Docker Compose setup involving Logstash. I've been following various tutorials to configure Logstash within an ELK (Elasticsearch, Logstash, Kibana) ...
Mehdi Ghasemi's user avatar
0 votes
0 answers
17 views

Data parse from multiple rsyslog to logstash to elasticsearch

I am new to ELK. I have deployed my ELK with this article. https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elastic-stack-on-ubuntu-22-04 I skipped the ...
milan patel's user avatar
0 votes
0 answers
10 views

Logs Pulling Architecture

Filebeat uses a "push" architecture, sending logs to logstash or elastic. Is there a feature or product that would enable elastic to receive hosts' logs using a "pull" architecture?...
Raz Maabari's user avatar
0 votes
0 answers
34 views

How to stream from Oracle Cloud Streaming to Elasticsearch Kafka Input Plugin?

I have set up Oracle Cloud Streaming all the way from pools to the stream itself configured as public, I am trying to use Logstash to pull the data and get it stored in Elasticsearch, and want to know ...
Zaid Afaneh's user avatar
0 votes
1 answer
69 views

Unable to configure plugins: (ArgumentError) Cannot determine timezone from nil\n(secs

[2023-08-21T20:43:58,795][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::...
Shi Eng Ng's user avatar
0 votes
0 answers
15 views

Docker logstash trying to connect to despite

I have a docker logstash configured as such: input { tcp { port => 5000 } } output { elasticsearch { hosts => ["https://my-custom-elastic.com:9321"] Expected: I would ...
PatPanda's user avatar
  • 3,686
0 votes
0 answers
27 views

error 'using JAVA_HOME while Logstash distribution comes with a bundled JDK \IBM was unexpected at this time.' when trying to start logstash on my pc

I'm trying to start logstash on my machine, but got below error C:\Users\xxx\Tool\logstash-7.13.3\bin>logstash -f ../test_cyrark_proxy.conf "test" Using JAVA_HOME defined java: "C:\...
ella's user avatar
  • 175
0 votes
1 answer
47 views

Logstash config doesn't parse JSON file and not push data to index - log: {:count=>1,:running_pipelines=>[:main], :non_running_pipelines=>[]}

I generate a JSON file every 5 mins through a Python code and try to pushing data to Elastic, but the Logstash throw following message and doesn't push any data to Kibana. My pipeline: File --> ...
M.A.Murali's user avatar
  • 9,998
1 vote
1 answer
43 views

Filebeat Processor isn't parsing field exception from json log file

I want to ingest JSON log files from an application into ElasticSearch. To do so, I am using Custom log files integration running on an Elastic agent, that is managed by a Fleet Manager. The log files ...
MiguelSlv's user avatar
  • 14.2k
0 votes
0 answers
71 views

Elasticsearch Logstash Kibana (ELK stack) + filebeat setup on ubuntu 22.04 AWS EC2 instance on t3.large to visualize system logs

I am trying to setup ELK stack + filebeat on ubuntu 22.04 AWS EC2 instance on t3.large to visualize system logs. below is the process i am following (Note - the steps are not copied from ChatGPT, the ...
Ansh Tyagi's user avatar
0 votes
0 answers
19 views

Logstash docker image - No logstash-plugin?

I have deployed LogStash on kubernetes using the latest image (logstash:8.8.1) and now I would like to install some plugins. I understand that I would need to use the 'logstash-plugin' tool, whis ...
Roel's user avatar
  • 1
0 votes
0 answers
14 views

can Logstash fetch data from 2 different sources in single Input JDBC section?

I am working on migrating logstash pipelines to read data from snowflake instead of sql. While most of my source data is available in snowflake, Couple tables are meant to be still in sql database as ...
user2506233's user avatar
0 votes
0 answers
17 views

add field in logstash with a stripped information from another field

Is there a valid function in logstash "add field" to strip a hostname to a part of it. i have variations of hostnames from a haproxy.log like this cccrz1-qa01.exserver.int cccrz2-qa01....
druuhl's user avatar
  • 1
0 votes
0 answers
11 views

Pausing Logstash During Synonym Updates

I'm currently engaged in a project that utilizes Elastic Search. Within this project, I've established a logstash pipeline that operates at 10-second intervals to ensure the synchronization of data ...
Rabi Siddique's user avatar
0 votes
0 answers
49 views

Masking nested objects in Spring boot using logstash-logback-encoder

I want to mask the password field of the log- {"timestamp":"2021-10-26T22:41:29.471+02:00","level":"INFO", "body":[{"id":1,"password&...
ALOK KUMAR's user avatar
1 vote
0 answers
51 views

Getting error : LOGBACK: No context given for net.logstash.logback.LogstashFormatter

I am trying to print a log in the SpringBoot Application using Logback and the following configuration, but the logs are not printing. Spring-boot version 2.7.2 and Java version 17 are being used. In ...
Ranjit Soni's user avatar
0 votes
1 answer
46 views

Does ECK required logstash

I'm quite confused between ECK and ELK. AFIK ELK requires logstash for reading the logs and whether the logstash required for ECK also or only filebeat or metric beat is enough to get logs. Filebeat ...
ilavarasan M's user avatar
0 votes
0 answers
32 views

Grok Pattern and Hostname Extraction Issues in Logstash 8.8.2

List item I am encountering an issue with Logstash 8 when processing logs from a client machine, while Logstash 6 correctly processes the same logs. Here are the log samples: Log Sample - Logstash 8 (...
Ts Ts's user avatar
  • 1
0 votes
0 answers
11 views

Secure Logstash and Filebeats communication

We are working on an integration where we need to take logs from Filebeat through Logstash. However, Filebeat and Logstash are hosted in different networks. In order to secure the communication, we ...
Seemant Bind's user avatar
0 votes
1 answer
34 views

Logstash how to send data from a file in a loop?

As part of load testing, I have a couple of premade log files that simulate different scenarios. I'm trying to use Logstash to send the content of 1 file, and resend it again in a loop every time it ...
dorony's user avatar
  • 1,018
0 votes
1 answer
39 views

ELK with Filebeat without Log File

I have setup my ELK stack with Filebeat to ship my logs, all works fine if I have a log file and add that path to my filebeat.yml But what if my application has no log files generated and only using ...
Lan Aaroon Jabasundar's user avatar
0 votes
1 answer
43 views

Logstash http filter with array fields in body

I want to create an HTTP Logstash filter that invokes an API using a request body that includes an array field. The input plugin reads a document as follows: { "tags": ["foo", &...
Ehsan Alemzadeh's user avatar
0 votes
0 answers
77 views

LogStash cannot connect Postgresql and ElasticSearch in docker-compose

I'm trying to run Elastic Stack on docker-compose and import data from postgresql, but LogStash cannot connect to postgresql and ElasticSearch. docker-compose.yml: version: '3' services: db: ...
lipsum's user avatar
  • 919
0 votes
0 answers
15 views

Logstash- How to parse formatted JSON arrays in log files

Logstash version: 7.17.10 Elasticsearch version:7.17.10 The logs are located in /var/logs directory and the format is as below: [ { "t": "SYS", "dt": "2023-...
fisherhe's user avatar

1
2 3 4 5
173