Splunk Search
Highlighted

Event breaking help with props.conf

Communicator

I have a sourcetype that I'm working with and trying to break up the events by any line that says "Job start time: yyyy-mm-dd hh:mm" then it ends. See example below. Still seeing events with multiple entries for this same regex line all in one event. Should be separating out into separate events. What am I missing?

PROPS.CONF
[legacyeft:eftpcl]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)^Job\sstart\stime:\s\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}$
NO_BINARY_CHECK=true
CHARSET=UTF-8
disabled=false

But still getting single events with 100+ lines in it, and several of those are the "Job start time..." lines that the regex should be separating into new events...

Results of (1) event; multiple entries for "Job start time..."
Job start time: 2019-10-29 13:02
... 79 lines omitted ...
Job start time: 2019-10-29 13:09
... 23 lines omitted ...
Job start time: 2019-10-29 13:39
... 23 lines omitted ...
Job start time: 2019-10-29 13:39
... 23 lines omitted ...
Job start time: 2019-10-29 13:02

0 Karma
Highlighted

Re: Event breaking help with props.conf

Communicator

also, the props.conf is on the cluster master under /opt/splunk/etc/master-apps/legacylinebreaker/local/props.conf and yes, it has been pushed to all the indexers in the cluster.

0 Karma
Highlighted

Re: Event breaking help with props.conf

Influencer

I suspect that with LINE_BREAKER, you should avoid the ^ and $ metacharacters, as those are for start and end of string. (yes in multi-line mode they match start and end of a line, but here where you're defining what is a line... I don't think they make sense.)

See no matches: https://regex101.com/r/mil3BJ/1

So, I would start with changing LINE_BREAKER to ([\r\n]+)Job\sstart\stime:\s\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}

Next the question is will you run into the default TRUNCATE length or not (are these events under 10000 characters long?) Does doing the LINEBREAKER method make sense for this data? (Related: I'd also define your TIMEPREFIX / TIME_FORMAT / TZ as appropriate.)

Additionally, are these events coming from a Universal Forwarder? If they're from a Heavy Forwarder your props need to go there as indexers won't re-parse already parsed events (by default... and there be dragons here... this is not black magic you really only want to get into until you really really need to)

View solution in original post

0 Karma
Highlighted

Re: Event breaking help with props.conf

Communicator

thanks for the reply acharlieh...

I will put your line breaker suggestions in place (mine arent working so...). Events are coming from a universal forwarder... but no heavy forwarder setup. So the props.conf is on the index cluster (as stated). I'll see if this fixes the issue!

Joe

0 Karma
Highlighted

Re: Event breaking help with props.conf

Contributor

and....did it?

0 Karma
Highlighted

Re: Event breaking help with props.conf

Communicator

Yes! It sure did. Always amazed and thankful for all the help. Awesome.

0 Karma