Splunk Search
Highlighted

Which is faster: SHOULD_LINEMERGE vs LINE_BREAKER

Path Finder

Hi, I'm doing some research for our new architecture and am currently doing some house keeping on our props and transforms.

If we have a regex for a sourcetype, is it faster to put the regex in the transforms or in the props and use linebreaker? Traditionally, we just set shouldlinemerge to false in the props and then put the regex in the transforms and link it via report in the props. However, I just read something that said doing regular expressions in line_breaker causes a performance increase.

Could someone provide a clear explanation on this for me?

0 Karma
Highlighted

Re: Which is faster: SHOULD_LINEMERGE vs LINE_BREAKER

SplunkTrust
SplunkTrust

Where did you read about regex in LINEBREAKER causing a performance increase?
FWIW, LINE
BREAKER defaults to the regex ([\r\n]+) so performance shouldn't be affected by regex.

---
If this reply helps you, an upvote would be appreciated.
0 Karma
Highlighted

Re: Which is faster: SHOULD_LINEMERGE vs LINE_BREAKER

Path Finder

NOTE: You get a significant boost to processing speed when you use
LINEBREAKER to delimit multi-line events (as opposed to using
SHOULD
LINEMERGE to reassemble individual lines into multi-line events).
* When using LINEBREAKER to delimit events, SHOULDLINEMERGE should be set
to false, to ensure no further combination of delimited events occurs.
http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf

0 Karma
Highlighted

Re: Which is faster: SHOULD_LINEMERGE vs LINE_BREAKER

Esteemed Legend

Using LINE_BREAKER= and SHOULD_LINEMERGE=false will always be WAAAAAAAY faster than using SHOULD_LINEMERGE=true. Obviously the better the RegEx in your LINE_BREAKER, the more efficient event processing will be so always spend extra time optimizing your LINE_BREAKER.

0 Karma
Highlighted

Re: Which is faster: SHOULD_LINEMERGE vs LINE_BREAKER

Path Finder

Thanks. I guess my question is could I put the regex in the line_break and not even need anything in the transforms. But, I'm going to use TRANSFORM-### to link to the transforms to ensure our data gets parsed at index time instead of search time. Thanks for letting me ponder.

View solution in original post

0 Karma