I have web server (Ubuntu) with Nginx + PHP.
It has Filebeat, which sends Nginx logs to Elastic ingestion node directly (no Logstash or anything else).
When I just installed it 1st time, I made some customizations to the pipeline, which Filebeat created.
Everything worked great for a month or so.
But I noticed, that every Filebeat upgrade result in the creation of new pipeline. Currently I have these:
filebeat-7.3.1-nginx-error-pipeline: {},
filebeat-7.4.1-nginx-error-pipeline: {},
filebeat-7.2.0-nginx-access-default: {},
filebeat-7.3.2-nginx-error-pipeline: {},
filebeat-7.4.1-nginx-access-default: {},
filebeat-7.3.1-nginx-access-default: {},
filebeat-7.3.2-nginx-access-default: {},
filebeat-7.2.0-nginx-error-pipeline: {}
I can create new pipeline, but how do I tell (how to configure) Filebeat to use specific pipeline?
Here is what I tried and it doesn't work:
- module: nginx
# Access logs
access:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
var.paths: ["/var/log/nginx/*/*access.log"]
# Convert the timestamp to UTC
var.convert_timezone: true
# The Ingest Node pipeline ID associated with this input. If this is set, it
# overwrites the pipeline option from the Elasticsearch output.
output.elasticsearch.pipeline: 'filebeat-nginx-access-default'
pipeline: 'filebeat-nginx-access-default
It still using filebeat-7.4.1-nginx-error-pipeline
pipeline.
Here is Filebeat instructions on how to configure it (but I can't make it work): https://github.com/elastic/beats/blob/7.4/filebeat/filebeat.reference.yml#L1129-L1130
Question: how can I configure Filebeat module to use specific pipeline?
Update (Nov 2019): I submitted related bug: https://github.com/elastic/beats/issues/14348