Splunk Search

Which is faster: SHOULD_LINEMERGE vs LINE_BREAKER

dkrichards16
Path Finder

Hi, I'm doing some research for our new architecture and am currently doing some house keeping on our props and transforms.

If we have a regex for a sourcetype, is it faster to put the regex in the transforms or in the props and use line_breaker? Traditionally, we just set should_linemerge to false in the props and then put the regex in the transforms and link it via report in the props. However, I just read something that said doing regular expressions in line_breaker causes a performance increase.

Could someone provide a clear explanation on this for me?

0 Karma
1 Solution

dkrichards16
Path Finder

Thanks. I guess my question is could I put the regex in the line_break and not even need anything in the transforms. But, I'm going to use TRANSFORM-### to link to the transforms to ensure our data gets parsed at index time instead of search time. Thanks for letting me ponder.

View solution in original post

0 Karma

dkrichards16
Path Finder

Thanks. I guess my question is could I put the regex in the line_break and not even need anything in the transforms. But, I'm going to use TRANSFORM-### to link to the transforms to ensure our data gets parsed at index time instead of search time. Thanks for letting me ponder.

0 Karma

woodcock
Esteemed Legend

Using LINE_BREAKER= and SHOULD_LINEMERGE=false will always be WAAAAAAAY faster than using SHOULD_LINEMERGE=true. Obviously the better the RegEx in your LINE_BREAKER, the more efficient event processing will be so always spend extra time optimizing your LINE_BREAKER.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Where did you read about regex in LINE_BREAKER causing a performance increase?
FWIW, LINE_BREAKER defaults to the regex ([\r\n]+) so performance shouldn't be affected by regex.

---
If this reply helps you, Karma would be appreciated.
0 Karma

dkrichards16
Path Finder

NOTE: You get a significant boost to processing speed when you use
LINE_BREAKER to delimit multi-line events (as opposed to using
SHOULD_LINEMERGE to reassemble individual lines into multi-line events).
* When using LINE_BREAKER to delimit events, SHOULD_LINEMERGE should be set
to false, to ensure no further combination of delimited events occurs.
http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf

Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...