topic Re: Parsing Multi Line Timestamp in Getting Data In

Parsing Multi Line Timestamp

jroedel — Tue, 01 Oct 2024 09:38:44 GMT

I have to parse the timestamp of JSON logs and I would like to include subsecond precision. My JSON-Events start like this:

{ "instant" : { "epochSecond" : 1727189281, "nanoOfSecond" : 202684061 }, ...

Thus I tried as config in props.conf:

TIME_FORMAT=%s,\n "nanoOfSecond" : %9N TIME_PREFIX="epochSecond" :\s MAX_TIMESTAMP_LOOKAHEAD=500

That did unfortunately not work.

What is the right way to parse this time stamp with subsecond precision?

Re: Parsing Multi Line Timestamp

gcusello — Tue, 01 Oct 2024 09:44:56 GMT

Hi @jroedel ,

are you sure about the number of spaces?

please try this:

TIME_FORMAT=%s,\n\s*"nanoOfSecond"\s*:\s*%9N TIME_PREFIX="epochSecond"\s*:\s* MAX_TIMESTAMP_LOOKAHEAD=500

Ciao.

Giuseppe

Re: Parsing Multi Line Timestamp

jroedel — Tue, 01 Oct 2024 09:47:18 GMT

I tried, but still no luck

Re: Parsing Multi Line Timestamp

gcusello — Tue, 01 Oct 2024 09:49:09 GMT

Hi @jroedel ,

please try this:

TIME_FORMAT=%s,\n\s*\"nanoOfSecond\"\s*:\s*%9N TIME_PREFIX=\"epochSecond\"\s*:\s* MAX_TIMESTAMP_LOOKAHEAD=500

Ciao.

Giuseppe

Re: Parsing Multi Line Timestamp

jroedel — Tue, 01 Oct 2024 09:54:36 GMT

Thanks for your second attempt. I tried, but still no luck.

Might there be the possibility, that the "Add Data" WebUI Wizard does not support this correctly?

Re: Parsing Multi Line Timestamp

gcusello — Tue, 01 Oct 2024 10:12:55 GMT

Hi @jroedel ,

if the Add Data feature doesn't permit to use this feature I suppose that it isn't possible event if it's strange.

I tried but I have the same result

Ciao.

Giuseppe

Re: Parsing Multi Line Timestamp

jroedel — Tue, 01 Oct 2024 13:45:17 GMT

Finally after a lot of testing I found a solution via transforms.conf

[timestamp-fix] INGEST_EVAL= _time=json_extract(_raw,"instant.epochSecond").".".json_extract(_raw,"instant.nanoOfSecond")

Furthermore, it turned out that regex is not allowed in TIME_FORMAT field in props.conf.

Re: Parsing Multi Line Timestamp

gcusello — Tue, 01 Oct 2024 14:07:18 GMT

Hi @jroedel ,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉