Splunk Search

Event breaking help with props.conf

joesrepsolc
Communicator

I have a sourcetype that I'm working with and trying to break up the events by any line that says "Job start time: yyyy-mm-dd hh:mm" then it ends. See example below. Still seeing events with multiple entries for this same regex line all in one event. Should be separating out into separate events. What am I missing?

PROPS.CONF
[legacyeft:eftpcl]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)^Job\sstart\stime:\s\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}$
NO_BINARY_CHECK=true
CHARSET=UTF-8
disabled=false

But still getting single events with 100+ lines in it, and several of those are the "Job start time..." lines that the regex should be separating into new events...

Results of (1) event; multiple entries for "Job start time..."
Job start time: 2019-10-29 13:02
... 79 lines omitted ...
Job start time: 2019-10-29 13:09
... 23 lines omitted ...
Job start time: 2019-10-29 13:39
... 23 lines omitted ...
Job start time: 2019-10-29 13:39
... 23 lines omitted ...
Job start time: 2019-10-29 13:02

0 Karma
1 Solution

acharlieh
Influencer

I suspect that with LINE_BREAKER, you should avoid the ^ and $ metacharacters, as those are for start and end of string. (yes in multi-line mode they match start and end of a line, but here where you're defining what is a line... I don't think they make sense.)

See no matches: https://regex101.com/r/mil3BJ/1

So, I would start with changing LINE_BREAKER to ([\r\n]+)Job\sstart\stime:\s\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}

Next the question is will you run into the default TRUNCATE length or not (are these events under 10000 characters long?) Does doing the LINE_BREAKER method make sense for this data? (Related: I'd also define your TIME_PREFIX / TIME_FORMAT / TZ as appropriate.)

Additionally, are these events coming from a Universal Forwarder? If they're from a Heavy Forwarder your props need to go there as indexers won't re-parse already parsed events (by default... and there be dragons here... this is not black magic you really only want to get into until you really really need to)

View solution in original post

0 Karma

joesrepsolc
Communicator

thanks for the reply acharlieh...

I will put your line breaker suggestions in place (mine arent working so...). Events are coming from a universal forwarder... but no heavy forwarder setup. So the props.conf is on the index cluster (as stated). I'll see if this fixes the issue!

Joe

0 Karma

darrenfuller
Contributor

and....did it?

0 Karma

joesrepsolc
Communicator

Yes! It sure did. Always amazed and thankful for all the help. Awesome.

0 Karma

acharlieh
Influencer

I suspect that with LINE_BREAKER, you should avoid the ^ and $ metacharacters, as those are for start and end of string. (yes in multi-line mode they match start and end of a line, but here where you're defining what is a line... I don't think they make sense.)

See no matches: https://regex101.com/r/mil3BJ/1

So, I would start with changing LINE_BREAKER to ([\r\n]+)Job\sstart\stime:\s\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}

Next the question is will you run into the default TRUNCATE length or not (are these events under 10000 characters long?) Does doing the LINE_BREAKER method make sense for this data? (Related: I'd also define your TIME_PREFIX / TIME_FORMAT / TZ as appropriate.)

Additionally, are these events coming from a Universal Forwarder? If they're from a Heavy Forwarder your props need to go there as indexers won't re-parse already parsed events (by default... and there be dragons here... this is not black magic you really only want to get into until you really really need to)

View solution in original post

0 Karma

joesrepsolc
Communicator

also, the props.conf is on the cluster master under /opt/splunk/etc/master-apps/legacy_line_breaker/local/props.conf and yes, it has been pushed to all the indexers in the cluster.

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!