0

Background

I am trying to override the default timestamp Graylog assigns messages with the one written in the message itself.

To achieve this I read the official docs:

https://docs.graylog.org/en/3.3/pages/extractors.html#the-standard-date-converter

However I am still unable to correctly set the timestamps.

Messages

Most of messages start like the following:

2021-02-14T06:07:58.521009+00:00 myApp00_rest_of_log

There are other types of messages that don't adhere to this standard, but that is for latter.

To extract the timestamp from the message I have created the following extractor:

extractor_screenshot

The RegEx does it's job nicely, however it's the converter that's killing it.

Problem

As you can see I am using the converter:

yyyy-MM-ddTHH:mm:ss.S

This doesn't work. I have also tried the following variations:

yyyy-MM-dd'T'HH:mm:ss.S
yyyy-MM-dd HH:mm:ss.S
yyyy-MM-dd'T'HH:mm:ss.SSSSSS
yyyy-MM-dd HH:mm:ss.SSSSSS
yyyy-MM-ddTHH:mm:ss.SSSSSS

At this point you might have picked up a couple of things:

  1. I have 2 problems: the "T" character in the middle and the precision that comes after the seconds (has 6 numbers, instead of just 3 (\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{6}))
  2. I am totally lost and I am just guessing random combinations at this point

OK but flawed solution

The "OK" solution would be to use the "Flexible Date" converter, however if I do that I will lose the precision that comes after the seconds (those 6 decimals mentioned in point 1).

Unfortunately for me, I need that amount of precision in my system, so I can't just cut out the last 6 decimals.

Questions

  1. How can I fix my converter so it can properly override the Timestamp with the precision I need?
4
  • According to: stackoverflow.com/questions/19223171/… The converter yyyy-MM-dd'T'HH:mm:ss.SSSSSS should work. However I am getting more errors in Graylog: java.lang.IllegalArgumentException: Invalid format: "2021-02-14T04:20:10.419248" is malformed at "248" Feb 16, 2021 at 16:50
  • Of course yyyy-MM-dd'T'HH:mm:ss.SSSSSS does not work. In the official documentation is written that S is a fraction of a second and also that it is in milliseconds. It is written in the presentation column. It seems that you can't achieve the microsecond precision using the standard date converter. Mar 4, 2021 at 2:15
  • @PetrFiedler is there a way to achieve microsecond precision in Graylog? Any way? Mar 5, 2021 at 8:58
  • @Flame_pheonix It does not seem so. Throughout the documentation, there are just milliseconds as the most precise unit. Mar 6, 2021 at 20:36

1 Answer 1

0

Answer

At the time of writing of this post, Graylog does NOT support microsecond precision in Timestamps.

This happens mainly because Graylog uses the org.joda.time.DateTime library, which does not support microsecond precision on timestamps.

There is currently a Pull request to try and fix this however:

https://github.com/Graylog2/graylog2-server/pull/9642

But it is unclear in which version of Graylog this will actually be released in.

For more information on the finding, feel free to check the official forum where the discovery was made:

https://community.graylog.org/t/extract-timestamp-from-message/18819/4?u=fl4m3ph03n1x

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.