Getting Data In

Why does one log contains multiple messages with same timestamp?

freddy_Guo
Path Finder

Hi guys,

I was wondering if some one could please give me a hand on this. We have written a custom TA to extract logs from a log source.

Log messages example:

 

 INFO 09 Feb 14:31:53 [pool-3-thread-1] WebHandlerAPI - Received GET request at /api/monitor/logger from [IP ADDRESS]
 INFO 09 Feb 14:31:53 [pool-4-thread-1] WebHandlerAPI - Received GET request at /api/monitor/performance from [IP ADDRESS]
 INFO 09 Feb 14:31:53 [thread_check] threadMonitor - 15 threads running OK

 

Props.conf

 

category = Application
disabled = false
pulldown_type = true
TIME_FORMAT = %d %b %T
TIME_PREFIX = \s+\w+\s+
MAX_TIMESTAMP_LOOKAHEAD = 20
EXTRACT-pdr_generic = (?:\s|)(?P<level>.*?)\s+(?P<timestamp>\d.*?)\s+\[(?P<message_type>.*?)\]\s+(?P<message>.*?)$

 

It would be great if someone could please point out which part of the props.conf needs to be improved.

 

 

Labels (1)
0 Karma
1 Solution

bowesmana
SplunkTrust
SplunkTrust

Add the following to props.comf

 

LINE_BREAKER = ([\r\n]+)
SHOULD_LINEMERGE = false

 

LINE_BREAKER is the default, but you the default for merge is true, so Splunk appears to be merging your lines

 

View solution in original post

0 Karma

bowesmana
SplunkTrust
SplunkTrust

You are relying on later information in the log/regex to constrain what the earlier part of the regex extracts, e.g.

(?P<timestamp>\d.*?)

is extracting a digit then anything up to the next part of the log, you would be better specifying a better timestamp extraction (e.g. (?P<timestamp>\d+\s\w{3}\s\d+:\d+:\d+)) that extracts what the timestamp should look like rather than hoping you will get the next space+square bracket to stop the extraction.

In general using non-greedy wildcards (.*?) to capture up to the next constraint is a bad idea.

I wasn't sure what the actual problem is when you say 'same timestamp'. There's no reason why you can't have log messages with the same timestamp - that's normal.

 

0 Karma

freddy_Guo
Path Finder

Hi @bowesmana ,

 

Thank you for the answer. 

I think I haven't been explained the question well.  If you see the screenshot below, the problem is that these logs are not breaking line properly. That's why I was thinking that it could be caused by props.conf.

6f9c4da1-0d68-4274-999c-acb485037452.jpg

8cde4c52-2bf3-4d51-b62f-b7ec9e97860a.jpg

0 Karma

bowesmana
SplunkTrust
SplunkTrust

Add the following to props.comf

 

LINE_BREAKER = ([\r\n]+)
SHOULD_LINEMERGE = false

 

LINE_BREAKER is the default, but you the default for merge is true, so Splunk appears to be merging your lines

 

0 Karma

freddy_Guo
Path Finder

Hi @bowesmana 

Thank you so much. 

It's working after I deploy it onto the indexer cluster. 

 

0 Karma

freddy_Guo
Path Finder

@bowesmana  

Hi thank you for the reply. 

I have added these two line on props.conf on universal forwarder as well as search heads cluster. It seems like we are still getting same error. 

freddy_Guo_1-1676501707292.png

 

 

0 Karma

bowesmana
SplunkTrust
SplunkTrust

Line breaking is done at index time, so having it on the search head will not affect how the data is ingested, so props.conf needs to have the settings on the indexer. I am not totally sure if line breaking occurs on the UF

 

freddy_Guo
Path Finder

Clicked on the wrong post to reply, but yeah. It came to me mind a bit too late. lol.

Thank you.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...