14

I have a drupal watchdog syslog file that I want to parse into essentially two nested fields, the syslog part and the message part so that I get this result

syslogpart: {
  timestamp: "",
  host: "",
  ...
},
messagepart:{
  parsedfield1: "",
  parsedfield2: "",
  ...
}

I tried making a custom pattern that looks like this:

DRUPALSYSLOG (%{SYSLOGTIMESTAMP:date} %{SYSLOGHOST:logsource} %{WORD:program}: %{URL:domain}\|%{EPOCH:epoch}\|%{WORD:instigator}\|%{IP:ip}\|%{URL:referrer}\|%{URL:request}\|(?<user_id>\d+)\|\|)

and then run match => ['message', '%{DRUPALSYSLOG:drupal}'}

but I don't get a nested response, I get a textblock drupal: "ALL THE MATCHING FIELDS IN ONE STRING" and then all the matches separately as well but not nested under drupal but rather on the same level.

2 Answers 2

29

Actually, you can do something like that in your pattern config

%{WORD:[drupal][program]}

It will create the json object like

drupal:{
  program: "..."
}
2
  • Yes, that does work except if you want to create subfields of @metadata (or, presumably, other fields with unusual characters in the name). Feb 17, 2017 at 16:40
  • Can confirm, still works with logstash-7.11.0. Example: APPLICATION (?:\[%{WORD:[application][context]}\])?\[%{WORD:[application][messageid]}\]\[%{WORD:[application][logcategory]}\]\[%{WORD:[application][logpriority]}\]
    – sastorsl
    Feb 15, 2021 at 14:27
12

Yes, this is expected. I don't think there's a way to produce nested fields with grok. I suspect you'll have to use the mutate filter to move them into place.

mutate {
    rename => {
      "date" => "[drupal][date]"
      "instigator" => "[drupal][instigator]"
      ...
    }
  }

If you have a lot of fields it might be more convenient to use a ruby filter. This is especially true if you prefix Drupal fields with e.g. "drupal." – then you'd write a filter to move all fields with that prefix into a subfield with the same name.

1
  • 1
    The bracket syntax is supported directly in the pattern definition. No need for mutate/rename! See Julien's answer below. Feb 16, 2017 at 10:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.