I am trying to configure Filebeats to index events into a custom-named index with a custom mapping for some of the fields. The problem is that Filebeat does not send events to my index but tries to send them to the default 'filebeats-xxx' index instead, and is failing with parsing/mapping exception since the events do not conform to the default filebeat event format (that's why I want to use my own mapping).
See details below.
UPDATE 09/10/2021: after finding a similar issue reported/discussed here: https://github.com/elastic/beats/issues/11866 I tried to follow the workaround and added the following parameter to the filebeats.yml to disable ILM in ES:
setup.ilm.enabled: false
This did not help much - filebeat logs were saying that ILM is already set. So I've added one more parameter:
setup.ilm.overwrite: true
and after this - the errors stopped, but when I try to push events through - filebeat seems to be hung after the following logs:
2021-09-09T21:42:58.057-0400 INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to backoff(elasticsearch(https://testbeats1.es.us-central1.gcp.cloud.es.io:9243))
2021-09-09T21:42:58.391-0400 INFO [esclientleg] eslegclient/connection.go:273 Attempting to connect to Elasticsearch version 7.14.1
2021-09-09T21:42:58.495-0400 INFO [publisher_pipeline_output] pipeline/output.go:151 Connection to backoff(elasticsearch(https://testbeats1.es.us-central1.gcp.cloud.es.io:9243)) established
I don't think it is a connectivity issue as the log says that a connection is established .... Any idea how to debug it further?
END of UPDATE
Here are the details:
I'm getting events from a GCP PubSup topic using the gcp-pubsub input type. The events are being received successfully (confirmed via using the Console output ) - wrapped into a lot of GCP/filebeat specific meta info. If I send an event like this to PubSub topic:
{
"referer": "https://blog.zoom.us/",
"activity_ip": "1.1.1.1",
"activity_date": "2021-09-08",
"logstash_id": "m_id_1",
"event_timestamp_millis": "1631126426000",
"cid": "1234567",
"ref_param": "https://www.yahoo.com",
"user_agent": "Mozilla/5.0 (X11; CrOS aarch64 13421.102.0) AppleWebKit/537.36 (KHTML, like Gecko)Chrome/86.0.4240.199 Safari/537.36"
}
the actual event received by Filebeats (and printed on the console when using the Console output) looks something like this:
{
"@timestamp": "2021-09-09T15:49:52.948Z",
"@metadata": {
"beat": "filebeat",
"type": "_doc",
"version": "7.14.1",
"_id": "59279bf715-2924539382735950"
},
"message": {
"referer": "https://blog.zoom.us/",
"activity_ip": "1.1.1.1",
"activity_date": "2021-09-08",
"logstash_id": "m_id_1",
"event_timestamp_millis": "1631126426000",
"cid": "1234567",
"ref_param": "https://www.yahoo.com",
"user_agent": "Mozilla/5.0 (X11; CrOS aarch64 13421.102.0) AppleWebKit/537.36 (KHTML, like Gecko)Chrome/86.0.4240.199 Safari/537.36"
},
"input": {
"type": "gcp-pubsub"
},
"ecs": {
"version": "1.10.0"
},
"host": {
"ip": [
"xxxx", ...
],
"mac": [
"...", ...
],
"hostname": "xxx.home",
"architecture": "x86_64",
"os": {
"platform": "darwin",
"version": "10.16",
"family": "darwin",
"name": "Mac OS X",
"kernel": "20.2.0",
"build": "20C69",
"type": "macos"
},
"name": "xxx",
"id": "AB905FFC-D83C-5512-B33A-AA1286026867"
},
"agent": {
"name": "xxx,
"type": "filebeat",
"version": "7.14.1",
"hostname": "xxx",...
},
"event": {
"id": "59279bf715-2924539382735950",
"created": "2021-09-09T15:49:52.936Z"
}
}
The actual event that I am sending is stored in the "message" element. By default, it comes as a String, but since I am sending JSON and want it to be treated as a JSON doc with fields/values - I have added the following Filebeats Processor to the filebeats.yml:
- decode_json_fields:
fields: ["message"]
(see the full filebeat.yml config below)
I also want to use my own types for the event fields - for example use 'ip' type for ip fields, date types for all date-specific fields , etc. This is why I wanted to use my own mapping for these events., and send them into an index with a name of my choosing - for example just for testing, it was 'marina-xxx':
The full mapping/template I came up with is specified below . At first, I tried putting it as a 'marina_es_template.json' file into the filebeat root dir and let Filebeat upload this mapping - so I tried the following config:
setup.template.name: "marina"
setup.template.pattern: "marina-*"
setup.template.json.enabled: true
setup.template.json.path: "marina_es_template.json"
setup.template.json.name: "marina"
setup.template.enabled: true
This did not work - filebeat was not uploading the template correctly - I could not find it in my ES (using GET _template/marina* command). So I have loaded it into my ES manually, using PUT command - and it worked fine, I can see the template listed in ES, and the content of the template is correct.
So, the final issue I was trying to solve was to make Filebeat actually send the events into a 'marina-test1' index and have the mapping applied correctly to this index - by matching on the index name pattern
I've added this config for ES output:
output.elasticsearch:
enabled: true
index: "marina-test2"
and used this template settings:
setup.template.name: "marina"
setup.template.pattern: "marina-*"
setup.template.json.enabled: false
And still Fielbeat keeps trying to send the incoming events into a 'filebeat-xxx' index and fails with the following error:
{"type":"mapper_parsing_exception","reason":"failed to parse field [message] of type [text] in document with id '59279bf715-2924626396675533'. ....}
which is correct, since in the 'filebeat' mapping the field 'message' is supposed to be a text, not an object ...
So, the question is: how do I force Filebeat to send events into my 'marina-xxx' index, and not 'filebeat-xxx' one???
Thank you!
Here is my filebeat.yml: (removed comments/unused fields for clarity
# ============================== Filebeat inputs ===============================
filebeat.inputs:
- type: gcp-pubsub
enabled: true
project_id: my-project
topic: logs-for-es-marina
subscription.name: logs-for-es-marina-sub
credentials_file: /mydir/.config/gcloud/application_default_credentials.json
fields_under_root: true
# ======================= Elasticsearch template setting =======================
setup.template.name: "marina"
setup.template.pattern: "marina-*"
setup.template.enabled: false
## also tried this - but did not help:
#setup.template.json.enabled: true
#setup.template.json.path: "marina_es_template.json"
#setup.template.json.name: "marina"
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
enabled: true
index: "marina-test2"
hosts: ["https://<my_es>:9243"]
username: "<my-user>"
password: "<my-pwd>"
# ================================= Processors =================================
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
- decode_json_fields:
fields: ["message"]
full ES template mapping:
PUT _template/marina
{
"index_patterns" :
["marina-*"],
"settings":{
"index": {
"number_of_shards":3,
"number_of_replicas":1
}
},
"mappings":{
"_source":{
"enabled":true
},
"properties" : {
"message": {
"properties":{
"logstash_id":{
"type":"keyword"
},
"event_timestamp_millis":{
"type":"date"
},
"activity_date":{
"type":"date",
"format":"yyyy-MM-dd"
},
"cid":{
"type":"keyword"
},
"activity_ip":{
"type":"ip"
},
"ref_param":{
"type":"text"
},
"referer":{
"type":"keyword"
},
"user_agent":{
"type":"text"
}
}
}
}
}
}