topic Re: Event Breaking using "Break_Only_Before" Log_level and Timestamp in Splunk Enterprise

Event Breaking

surajsplunkd — Thu, 20 Jul 2023 04:17:27 GMT

Hello Everyone, I have tried multiple times but i am unable to break event before the log_level(INFO and WARNING) as in below logs.

Could you please help me break below logs into events starting with log_level?

Re: Event Breaking using "Break_Only_Before" Log_level and Timestamp

richgalloway — Wed, 12 Jul 2023 13:51:19 GMT

Try these props:

Re: Event Breaking using "Break_Only_Before" Log_level and Timestamp

surajsplunkd — Wed, 12 Jul 2023 14:41:28 GMT

Hi @richgalloway

many thanks it works. I also came up with below regexes which works as expected. Putting below for your reference.

However, would you be able to help me understand as to why with the same props settings, most of the raw data is breaking into events at log_level followed by timestamps as expected while some are not.

Immediate help is highly appreciated.

Not working

Working

Together

Thanks in advance

Re: Event Breaking using "Break_Only_Before" Log_level and Timestamp

richgalloway — Wed, 12 Jul 2023 14:39:45 GMT

I suspect the events not breaking correctly do not have white space before the log level. The regex I provided uses an empty capture group to put the event break before the log level. If there's risk of a log level keyword being elsewhere in an event then add ":" on the end of the LINE_BREAKER setting to ensure it only matches the log level value at the beginning of the event.

Re: Event Breaking using "Break_Only_Before" Log_level and Timestamp

surajsplunkd — Wed, 12 Jul 2023 15:23:23 GMT

Appreciate your help @richgalloway , tested with the suggested changes, it was not helpful though.

In notepad, it looks clean same as working lines, however not at all able to figure out cause of this behavior.

First para represents non breaking lines

Second para represents breaking lines

Re: Event Breaking using "Break_Only_Before" Log_level and Timestamp

richgalloway — Wed, 12 Jul 2023 16:23:34 GMT

I'm at a loss here. The two paragraphs appear the same to me so I don't know why the behavior is different.

Re: Event Breaking using "Break_Only_Before" Log_level and Timestamp

surajsplunkd — Thu, 20 Jul 2023 03:27:01 GMT

I think i should check raw data in the above snip labelled as together on the server where it's getting generated couz , if you see the merged events they are exactly same. That is the first thing and secondly, if you would notice timestamps of the merged lines in merged events , they are chronological. If I am correct it should be reverse chronological from top to bottom. I therefore, think those merged lines are not separate lines with timestamps rather they are part of the very first line and Probably that is why splunk is putting them all together into one event with timestamps in increasing order. Well if that is so then how could the next unique event above the merged one have smaller timestamp? And that is why I suppose I need to check the order of lines with timestamps in rawdata on server itself.

Furthermore, why are there duplicate merged events. That is another question to be answered. 😁

Please correct me if i am wrong.

Re: Event Breaking using "Break_Only_Before" Log_level and Timestamp

richgalloway — Wed, 12 Jul 2023 19:01:16 GMT

Yes, Splunk displays events in reverse chronological order by default That you see incrementing rather than decrementing timestamps is a symptom of failed line breaking and not a cause of it. Once we get Splunk to break events properly, all events will be in the proper time sequence.

Re: Event Breaking using "Break_Only_Before" Log_level and Timestamp

surajsplunkd — Thu, 20 Jul 2023 03:32:24 GMT

Later, I noticed, SHOULD_LINEMERGE=false was missing and adding this resolved my issue . @richgalloway appreciate your quick response and resolution.

SHOULD_LINEMERGE=false