Why does one log contains multiple messages with same timestamp?

[freddy_Guo]

Hi guys,

I was wondering if some one could please give me a hand on this. We have written a custom TA to extract logs from a log source.

Log messages example:

 INFO 09 Feb 14:31:53 [pool-3-thread-1] WebHandlerAPI - Received GET request at /api/monitor/logger from [IP ADDRESS]
 INFO 09 Feb 14:31:53 [pool-4-thread-1] WebHandlerAPI - Received GET request at /api/monitor/performance from [IP ADDRESS]
 INFO 09 Feb 14:31:53 [thread_check] threadMonitor - 15 threads running OK

Props.conf

category = Application
disabled = false
pulldown_type = true
TIME_FORMAT = %d %b %T
TIME_PREFIX = \s+\w+\s+
MAX_TIMESTAMP_LOOKAHEAD = 20
EXTRACT-pdr_generic = (?:\s|)(?P<level>.*?)\s+(?P<timestamp>\d.*?)\s+\[(?P<message_type>.*?)\]\s+(?P<message>.*?)$

It would be great if someone could please point out which part of the props.conf needs to be improved.

[bowesmana]

You are relying on later information in the log/regex to constrain what the earlier part of the regex extracts, e.g.

(?P<timestamp>\d.*?)

is extracting a digit then anything up to the next part of the log, you would be better specifying a better timestamp extraction (e.g. (?P<timestamp>\d+\s\w{3}\s\d+:\d+:\d+)) that extracts what the timestamp should look like rather than hoping you will get the next space+square bracket to stop the extraction.

In general using non-greedy wildcards (.*?) to capture up to the next constraint is a bad idea.

I wasn't sure what the actual problem is when you say 'same timestamp'. There's no reason why you can't have log messages with the same timestamp - that's normal.

[freddy_Guo]

Hi @bowesmana ,

Thank you for the answer. 

I think I haven't been explained the question well.  If you see the screenshot below, the problem is that these logs are not breaking line properly. That's why I was thinking that it could be caused by props.conf.

[bowesmana · Accepted solution]

Add the following to props.comf

LINE_BREAKER = ([\r\n]+)
SHOULD_LINEMERGE = false

LINE_BREAKER is the default, but you the default for merge is true, so Splunk appears to be merging your lines

[freddy_Guo]

Hi @bowesmana 

Thank you so much. 

It's working after I deploy it onto the indexer cluster. 

[freddy_Guo]

@bowesmana  

Hi thank you for the reply. 

I have added these two line on props.conf on universal forwarder as well as search heads cluster. It seems like we are still getting same error. 

[bowesmana]

Line breaking is done at index time, so having it on the search head will not affect how the data is ingested, so props.conf needs to have the settings on the indexer. I am not totally sure if line breaking occurs on the UF

[freddy_Guo]

Clicked on the wrong post to reply, but yeah. It came to me mind a bit too late. lol.

Thank you.