Event Breaking

[surajsplunkd]

Hello Everyone, I have tried multiple times but i am unable to break event before the log_level(INFO and WARNING) as in below logs.

Could you please help me break below logs into events starting with log_level?

[surajsplunkd]

Later, I noticed, SHOULD_LINEMERGE=false was missing and adding this resolved my issue . @richgalloway  appreciate your quick response and resolution.

SHOULD_LINEMERGE=false

[surajsplunkd]

Hi @richgalloway 

many thanks it works. I also came up with below regexes which works as expected. Putting below for your reference.

(\s+)(ERROR|WARNING|WARN|DEBUG|TRACE|INFO)
(\s+)(INFO|WARNING|ERROR|WARN|DEBUG|TRACE).*.\d+:\d+:\d+

However, would you be able to help me understand as to why with the same props settings, most of the raw data is breaking into events at log_level followed by timestamps as expected while some are not. 

Immediate help is highly appreciated.

Not working

Working 

Together

Thanks in advance

[richgalloway]

I suspect the events not breaking correctly do not have white space before the log level.  The regex I provided uses an empty capture group to put the event break before the log level.  If there's risk of a log level keyword being elsewhere in an event then add ":" on the end of the LINE_BREAKER setting to ensure it only matches the log level value at the beginning of the event.

[surajsplunkd]

Appreciate your help @richgalloway , tested with the suggested changes, it was not helpful though.

In notepad, it looks clean same as working lines, however not at all able to figure out cause of this behavior.

First para represents non breaking lines

Second para represents breaking lines

[richgalloway]

I'm at a loss here.  The two paragraphs appear the same to me so I don't know why the behavior is different.

[surajsplunkd]

I think i should check raw data in the above snip labelled as together on the server where it's getting generated couz , if you see the merged events they are exactly same. That is the first thing and secondly, if you would notice timestamps of the merged lines in merged events , they are chronological. If I am correct it should be reverse chronological from top to bottom. I therefore, think those merged lines are not separate lines with timestamps rather they are part of the very first line and Probably that is why splunk is putting them all together into one event with timestamps in increasing order. Well if that is so then how could the next unique event above the merged one have smaller timestamp? And that is why I suppose I need to check the order of lines with timestamps in rawdata on server itself.

Furthermore, why are there duplicate merged events. That is another question to be answered. 😁

Please correct me if i am wrong. 

[richgalloway]

Yes, Splunk displays events in reverse chronological order by default  That you see incrementing rather than decrementing timestamps is a symptom of failed line breaking and not a cause of it.  Once we get Splunk to break events properly, all events will be in the proper time sequence.

[richgalloway · Accepted solution]

Try these props:

[mysourcetype]
SHOULD_LINEMERGE=false
LINE_BREAKER=()(INFO|WARNING|ERROR|TRACE|DEBUG)
TIME_FORMAT=%b %d %H:%M:%S
TIME_PREFIX=(INFO|WARNING|ERROR|TRACE|DEBUG):