1

i'm using elk stack of version 5.5 in ubuntu 16.0

My logs are getting broken and not writting properly into elastic which is resulting in json.erros

like below

Error decoding JSON: invalid character 'e' in literal null (expecting 'u')"  

getting json.errors very frequent and those logs are not reading or writting properly into elasticsearch ?

and this is happening for every 5 to 10 mins. please help me solve it.

screenshot of broken logs in kibana

My sample log is :

{"log":"2019-10-01 07:18:26:854*[DEBUG]*cluster2-nio-worker-0*Connection*userEventTriggered*Connection[cassandraclient/10.3.254.137:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat\n","stream":"stdout","time":"2019-10-01T07:18:26.85462769Z"}

9
  • Yeah, this has been fixed but you have to upgade es version. Take a look at: github.com/elastic/beats/issues/6045
    – LeBigCat
    Oct 9, 2019 at 8:11
  • Can you please share the filebeat prospector configuration for that particular file? And also: Are your JSON objects in the log file pretty-printed, meaning they span over multiple lines? Oct 9, 2019 at 9:24
  • Thanks for the reply @apt-get_install_skill The following is my filebeat configuration : filebeat.registry_file: /var/log/containers/filebeat_registry filebeat.inputs: - type: log paths: - "/var/log/containers/*.log" tags: ["kube-logs"] symlinks: true json.message_key: log json.keys_under_root: true json.add_error_key: true multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}' multiline.negate: true multiline.match: after Oct 9, 2019 at 10:35
  • Are your JSON objects in the log file pretty-printed, meaning they span over multiple lines? Oct 9, 2019 at 10:58
  • 1
    @apt-get_install_skill i already did but it showing the following pop : Votes cast by those with less than 15 reputation are recorded, but do not change the publicly displayed post score. Oct 23, 2019 at 5:38

1 Answer 1

0

Since you have stated that the json logs are not pretty printed I assume that the multiline-settings of your input configuration are causing the problems.

In my opinion you don't need any multiline settings when you have logs in json format and they are not pretty printed, meaning the whole json object (= log event) is written in one line.

You have already specified

json.message_key: log

This solely should get the job done.

So to sum it up:

Remove the multiline settings and try again. Your configuration should look like this:

filebeat.inputs: 
  - type: log 
    paths: 
      - "/var/log/containers/*.log"
    tags: ["kube-logs"]
    symlinks: true 
    json.message_key: log
    json.keys_under_root: true 
    json.add_error_key: true
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.