Splunk Search

Which is faster: SHOULD_LINEMERGE vs LINE_BREAKER

dkrichards16
Path Finder

Hi, I'm doing some research for our new architecture and am currently doing some house keeping on our props and transforms.

If we have a regex for a sourcetype, is it faster to put the regex in the transforms or in the props and use line_breaker? Traditionally, we just set should_linemerge to false in the props and then put the regex in the transforms and link it via report in the props. However, I just read something that said doing regular expressions in line_breaker causes a performance increase.

Could someone provide a clear explanation on this for me?

0 Karma
1 Solution

dkrichards16
Path Finder

Thanks. I guess my question is could I put the regex in the line_break and not even need anything in the transforms. But, I'm going to use TRANSFORM-### to link to the transforms to ensure our data gets parsed at index time instead of search time. Thanks for letting me ponder.

View solution in original post

0 Karma

dkrichards16
Path Finder

Thanks. I guess my question is could I put the regex in the line_break and not even need anything in the transforms. But, I'm going to use TRANSFORM-### to link to the transforms to ensure our data gets parsed at index time instead of search time. Thanks for letting me ponder.

0 Karma

woodcock
Esteemed Legend

Using LINE_BREAKER= and SHOULD_LINEMERGE=false will always be WAAAAAAAY faster than using SHOULD_LINEMERGE=true. Obviously the better the RegEx in your LINE_BREAKER, the more efficient event processing will be so always spend extra time optimizing your LINE_BREAKER.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Where did you read about regex in LINE_BREAKER causing a performance increase?
FWIW, LINE_BREAKER defaults to the regex ([\r\n]+) so performance shouldn't be affected by regex.

---
If this reply helps you, Karma would be appreciated.
0 Karma

dkrichards16
Path Finder

NOTE: You get a significant boost to processing speed when you use
LINE_BREAKER to delimit multi-line events (as opposed to using
SHOULD_LINEMERGE to reassemble individual lines into multi-line events).
* When using LINE_BREAKER to delimit events, SHOULD_LINEMERGE should be set
to false, to ensure no further combination of delimited events occurs.
http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf

Get Updates on the Splunk Community!

Get ready to show some Splunk Certification swagger at .conf24!

Dive into the deep end of data by earning a Splunk Certification at .conf24. We're enticing you again this ...

Built-in Service Level Objectives Management to Bridge the Gap Between Service & ...

Now On-Demand Join us to learn more about how you can leverage Service Level Objectives (SLOs) and the new ...

Database Performance Sidebar Panel Now on APM Database Query Performance & Service ...

We’ve streamlined the troubleshooting experience for database-related service issues by adding a database ...