Getting Data In

date parsing and linemerge

Sqig
Path Finder

Hi folks.

We have an entry in props to parse our custom datestamps (format is YYYYMMDD HHMMSS.nnn) as follows:

  MAX_TIMESTAMP_LOOKAHEAD = 50
  NO_BINARY_CHECK = 1
  SHOULD_LINEMERGE = false
  TIME_FORMAT = %Y%m%d %H%M%S.%3N
  TIME_PREFIX = ^
  pulldown_type = 1

Quite often, our Splunk heavy forwarder reports that it cannot parse timestamps for these sourcetypes. What I see in the source logfiles is that sometimes, our logs do contain multiple lines (in the form of Java errors)... In that case, should we change the SHOULD_LINEMERGE from no to yes? Or do we need a far more complex regex to indicate the start of a new line?

Example (seriously munged and truncated, but just to give you an idea):

0130805 160141.074 some message about an error
com.stuff.api. Transactior timedout
        at com.stuff.exception
        at com.stuff.exception
        at com.stuff.exception
        at com.stuff
        at com.stuff
        at com.stuff
0 Karma
1 Solution

Ayn
Legend

If you don't want Splunk to get confused as to what timestamp it should set for the subsequent lines in multiline events, you need to set SHOULD_LINEMERGE to yes. The very fact that you're talking about that logs contain multiple lines supports this. Without SHOULD_LINEMERGE set to yes, Splunk will treat each line as an individual event (obviously) which you won't want unless you know what you're doing and why.

View solution in original post

0 Karma

Ayn
Legend

If you don't want Splunk to get confused as to what timestamp it should set for the subsequent lines in multiline events, you need to set SHOULD_LINEMERGE to yes. The very fact that you're talking about that logs contain multiple lines supports this. Without SHOULD_LINEMERGE set to yes, Splunk will treat each line as an individual event (obviously) which you won't want unless you know what you're doing and why.

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...