Event breaking help with props.conf

[joesrepsolc]

I have a sourcetype that I'm working with and trying to break up the events by any line that says "Job start time: yyyy-mm-dd hh:mm" then it ends. See example below. Still seeing events with multiple entries for this same regex line all in one event. Should be separating out into separate events. What am I missing?

PROPS.CONF
[legacyeft:eftpcl]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)^Job\sstart\stime:\s\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}$
NO_BINARY_CHECK=true
CHARSET=UTF-8
disabled=false

But still getting single events with 100+ lines in it, and several of those are the "Job start time..." lines that the regex should be separating into new events...

Results of (1) event; multiple entries for "Job start time..."
Job start time: 2019-10-29 13:02
... 79 lines omitted ...
Job start time: 2019-10-29 13:09
... 23 lines omitted ...
Job start time: 2019-10-29 13:39
... 23 lines omitted ...
Job start time: 2019-10-29 13:39
... 23 lines omitted ...
Job start time: 2019-10-29 13:02

[joesrepsolc]

thanks for the reply acharlieh...

I will put your line breaker suggestions in place (mine arent working so...). Events are coming from a universal forwarder... but no heavy forwarder setup. So the props.conf is on the index cluster (as stated). I'll see if this fixes the issue!

Joe

[darrenfuller]

and....did it?

[joesrepsolc]

Yes! It sure did. Always amazed and thankful for all the help. Awesome.

[acharlieh · Accepted solution]

I suspect that with LINE_BREAKER, you should avoid the ^ and $ metacharacters, as those are for start and end of string. (yes in multi-line mode they match start and end of a line, but here where you're defining what is a line... I don't think they make sense.)

See no matches: https://regex101.com/r/mil3BJ/1

So, I would start with changing LINE_BREAKER to ([\r\n]+)Job\sstart\stime:\s\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}

Next the question is will you run into the default TRUNCATE length or not (are these events under 10000 characters long?) Does doing the LINE_BREAKER method make sense for this data? (Related: I'd also define your TIME_PREFIX / TIME_FORMAT / TZ as appropriate.)

Additionally, are these events coming from a Universal Forwarder? If they're from a Heavy Forwarder your props need to go there as indexers won't re-parse already parsed events (by default... and there be dragons here... this is not black magic you really only want to get into until you really really need to)

[joesrepsolc]

also, the props.conf is on the cluster master under /opt/splunk/etc/master-apps/legacy_line_breaker/local/props.conf and yes, it has been pushed to all the indexers in the cluster.