topic Re: One log contains multiple messages with same timestamp in Getting Data In

Why does one log contains multiple messages with same timestamp?

freddy_Guo — Thu, 09 Feb 2023 16:48:39 GMT

Hi guys,

I was wondering if some one could please give me a hand on this. We have written a custom TA to extract logs from a log source.

Log messages example:

INFO 09 Feb 14:31:53 [pool-3-thread-1] WebHandlerAPI - Received GET request at /api/monitor/logger from [IP ADDRESS] INFO 09 Feb 14:31:53 [pool-4-thread-1] WebHandlerAPI - Received GET request at /api/monitor/performance from [IP ADDRESS] INFO 09 Feb 14:31:53 [thread_check] threadMonitor - 15 threads running OK

Props.conf

category = Application disabled = false pulldown_type = true TIME_FORMAT = %d %b %T TIME_PREFIX = \s+\w+\s+ MAX_TIMESTAMP_LOOKAHEAD = 20 EXTRACT-pdr_generic = (?:\s|)(?P<level>.*?)\s+(?P<timestamp>\d.*?)\s+\[(?P<message_type>.*?)\]\s+(?P<message>.*?)$

It would be great if someone could please point out which part of the props.conf needs to be improved.

Re: One log contains multiple messages with same timestamp

bowesmana — Thu, 09 Feb 2023 05:34:55 GMT

You are relying on later information in the log/regex to constrain what the earlier part of the regex extracts, e.g.

(?P<timestamp>\d.*?)

is extracting a digit then anything up to the next part of the log, you would be better specifying a better timestamp extraction (e.g. (?P<timestamp>\d+\s\w{3}\s\d+:\d+:\d+)) that extracts what the timestamp should look like rather than hoping you will get the next space+square bracket to stop the extraction.

In general using non-greedy wildcards (.*?) to capture up to the next constraint is a bad idea.

I wasn't sure what the actual problem is when you say 'same timestamp'. There's no reason why you can't have log messages with the same timestamp - that's normal.

Re: One log contains multiple messages with same timestamp

freddy_Guo — Thu, 09 Feb 2023 05:52:15 GMT

Hi @bowesmana ,

Thank you for the answer.

I think I haven't been explained the question well. If you see the screenshot below, the problem is that these logs are not breaking line properly. That's why I was thinking that it could be caused by props.conf.

Re: One log contains multiple messages with same timestamp

bowesmana — Fri, 10 Feb 2023 00:29:53 GMT

Add the following to props.comf

LINE_BREAKER = ([\r\n]+) SHOULD_LINEMERGE = false

LINE_BREAKER is the default, but you the default for merge is true, so Splunk appears to be merging your lines

Re: One log contains multiple messages with same timestamp

freddy_Guo — Wed, 15 Feb 2023 22:55:31 GMT

@bowesmana

Hi thank you for the reply.

I have added these two line on props.conf on universal forwarder as well as search heads cluster. It seems like we are still getting same error.

Re: One log contains multiple messages with same timestamp

bowesmana — Wed, 15 Feb 2023 23:16:30 GMT

Line breaking is done at index time, so having it on the search head will not affect how the data is ingested, so props.conf needs to have the settings on the indexer. I am not totally sure if line breaking occurs on the UF

Re: One log contains multiple messages with same timestamp

freddy_Guo — Wed, 15 Feb 2023 23:17:27 GMT

Hi @bowesmana

Thank you so much.

It's working after I deploy it onto the indexer cluster.

Re: One log contains multiple messages with same timestamp

freddy_Guo — Wed, 15 Feb 2023 23:21:44 GMT

Clicked on the wrong post to reply, but yeah. It came to me mind a bit too late. lol.

Thank you.