Getting Data In
Provide Splunk Cloud feedback in this confidential UX survey by June 17
for a chance to win a $200 Amazon gift card!

Default LINE_BREAKER broken?

cwilmoth
Path Finder

We recently upgraded from 6.3.3 to 6.4.1 in an attempt to fix some performance issues. After upgrading, there were a ton of "Changing breaking behavior for event stream because MAX_EVENTS (256) was exceeded without a single event break..." for multiple data sources on our heavy forwarders. I struggled to figure out why and eventually just created a [default] stanza in the props.conf file that gets deployed to both of our heavy forwarders and put the default LINE_BREAKER = ([\r\n]+) in there. After deployment, events are breaking just fine (like they were before).

Is this a known issue? I did not see anything in the release notes.

Thanks.

Tags (1)

martin_mueller
SplunkTrust
SplunkTrust

There's a second change, the without list has should linemerge set to true while the with list has it set to false. This tells Splunk to merge lines back together to whole events after applying the line breaker. Try setting should linemerge to false without setting the line breaker.

0 Karma

jmallorquin
Builder

Hi,

The problem is that you configured the F:\Splunk\etc\system\default\props.conf SHOULD_LINEMERGE in the default directory.
You should never change the configuration in this directory becouse when you upgrade splunk overwrite default files.

Hope i help you

0 Karma

cwilmoth
Path Finder

Nothing has been changed in the default directory. The props.conf file is dated 5/12/2016 just like all the other default files that were put in place by the 6.4.1 upgrade. The previous default files (6.3.3) were all dated 4/28/2015 and that old props.conf file also had SHOULD_LINEMERGE set to true.

From props.conf.spec:

SHOULD_LINEMERGE = [true|false]
* When set to true, Splunk combines several lines of data into a single
multiline event, based on the following configuration attributes.
* *Defaults to true.
*

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Run $SPLUNK_HOME/bin/splunk cmd btool --debug props list that_sourcetype with and without the extra default stanza and compare the output.

0 Karma

cwilmoth
Path Finder

With:
F:\Splunk\etc\apps\Dso_deploy_hvy_fwdrs\default\props.conf [deepsecurity-system_events]
F:\Splunk\etc\system\default\props.conf ANNOTATE_PUNCT = True
F:\Splunk\etc\system\default\props.conf AUTO_KV_JSON = true
F:\Splunk\etc\system\default\props.conf BREAK_ONLY_BEFORE =
F:\Splunk\etc\system\default\props.conf BREAK_ONLY_BEFORE_DATE = True
F:\Splunk\etc\system\default\props.conf CHARSET = AUTO
F:\Splunk\etc\system\default\props.conf DATETIME_CONFIG = \etc\datetime.xml
F:\Splunk\etc\system\default\props.conf HEADER_MODE =
F:\Splunk\etc\system\default\props.conf LEARN_SOURCETYPE = true
F:\Splunk\etc\apps\Dso_deploy_hvy_fwdrs\default\props.conf LINE_BREAKER = ([\r\n]+)
F:\Splunk\etc\system\default\props.conf LINE_BREAKER_LOOKBEHIND = 100
F:\Splunk\etc\system\local\props.conf MAX_DAYS_AGO = 90
F:\Splunk\etc\system\default\props.conf MAX_DAYS_HENCE = 2
F:\Splunk\etc\system\default\props.conf MAX_DIFF_SECS_AGO = 3600
F:\Splunk\etc\system\default\props.conf MAX_DIFF_SECS_HENCE = 604800
F:\Splunk\etc\system\default\props.conf MAX_EVENTS = 256
F:\Splunk\etc\system\default\props.conf MAX_TIMESTAMP_LOOKAHEAD = 128
F:\Splunk\etc\system\default\props.conf MUST_BREAK_AFTER =
F:\Splunk\etc\system\default\props.conf MUST_NOT_BREAK_AFTER =
F:\Splunk\etc\system\default\props.conf MUST_NOT_BREAK_BEFORE =
F:\Splunk\etc\system\default\props.conf SEGMENTATION = indexing
F:\Splunk\etc\system\default\props.conf SEGMENTATION-all = full
F:\Splunk\etc\system\default\props.conf SEGMENTATION-inner = inner
F:\Splunk\etc\system\default\props.conf SEGMENTATION-outer = outer
F:\Splunk\etc\system\default\props.conf SEGMENTATION-raw = none
F:\Splunk\etc\system\default\props.conf SEGMENTATION-standard = standard
F:\Splunk\etc\apps\Dso_deploy_hvy_fwdrs\default\props.conf SHOULD_LINEMERGE = false
F:\Splunk\etc\system\default\props.conf TRANSFORMS =
F:\Splunk\etc\apps\rb_steelhead_ta\default\props.conf TRANSFORMS-riverbed_src = riverbed_src
F:\Splunk\etc\apps\Dso_deploy_hvy_fwdrs\default\props.conf TRANSFORMS-t3 = set-tm-fw-sourcetype,set-tm-log-sourcetype,set-tm-im-sourcetype,set-tm-ip-sourcetype,set-tm-ipsevents
F:\Splunk\etc\system\default\props.conf TRUNCATE = 10000
F:\Splunk\etc\system\default\props.conf detect_trailing_nulls = auto
F:\Splunk\etc\system\default\props.conf maxDist = 100
F:\Splunk\etc\system\default\props.conf priority =
F:\Splunk\etc\system\default\props.conf sourcetype =

Without:
F:\Splunk\etc\apps\Dso_deploy_hvy_fwdrs\default\props.conf [deepsecurity-system_events]
F:\Splunk\etc\system\default\props.conf ANNOTATE_PUNCT = True
F:\Splunk\etc\system\default\props.conf AUTO_KV_JSON = true
F:\Splunk\etc\system\default\props.conf BREAK_ONLY_BEFORE =
F:\Splunk\etc\system\default\props.conf BREAK_ONLY_BEFORE_DATE = True
F:\Splunk\etc\system\default\props.conf CHARSET = AUTO
F:\Splunk\etc\system\default\props.conf DATETIME_CONFIG = \etc\datetime.xml
F:\Splunk\etc\system\default\props.conf HEADER_MODE =
F:\Splunk\etc\system\default\props.conf LEARN_SOURCETYPE = true
F:\Splunk\etc\system\default\props.conf LINE_BREAKER_LOOKBEHIND = 100
F:\Splunk\etc\system\local\props.conf MAX_DAYS_AGO = 90
F:\Splunk\etc\system\default\props.conf MAX_DAYS_HENCE = 2
F:\Splunk\etc\system\default\props.conf MAX_DIFF_SECS_AGO = 3600
F:\Splunk\etc\system\default\props.conf MAX_DIFF_SECS_HENCE = 604800
F:\Splunk\etc\system\default\props.conf MAX_EVENTS = 256
F:\Splunk\etc\system\default\props.conf MAX_TIMESTAMP_LOOKAHEAD = 128
F:\Splunk\etc\system\default\props.conf MUST_BREAK_AFTER =
F:\Splunk\etc\system\default\props.conf MUST_NOT_BREAK_AFTER =
F:\Splunk\etc\system\default\props.conf MUST_NOT_BREAK_BEFORE =
F:\Splunk\etc\system\default\props.conf SEGMENTATION = indexing
F:\Splunk\etc\system\default\props.conf SEGMENTATION-all = full
F:\Splunk\etc\system\default\props.conf SEGMENTATION-inner = inner
F:\Splunk\etc\system\default\props.conf SEGMENTATION-outer = outer
F:\Splunk\etc\system\default\props.conf SEGMENTATION-raw = none
F:\Splunk\etc\system\default\props.conf SEGMENTATION-standard = standard
F:\Splunk\etc\system\default\props.conf SHOULD_LINEMERGE = True
F:\Splunk\etc\system\default\props.conf TRANSFORMS =
F:\Splunk\etc\apps\rb_steelhead_ta\default\props.conf TRANSFORMS-riverbed_src = riverbed_src
F:\Splunk\etc\apps\Dso_deploy_hvy_fwdrs\default\props.conf TRANSFORMS-t3 = set-tm-fw-sourcetype,set-tm-log-sourcetype,set-tm-im-sourcetype,set-tm-ip-sourcetype,set-tm-ipsevents
F:\Splunk\etc\system\default\props.conf TRUNCATE = 10000
F:\Splunk\etc\system\default\props.conf detect_trailing_nulls = auto
F:\Splunk\etc\system\default\props.conf maxDist = 100
F:\Splunk\etc\system\default\props.conf priority =
F:\Splunk\etc\system\default\props.conf sourcetype =

0 Karma
Take the 2021 Splunk Career Survey

Help us learn about how Splunk has
impacted your career by taking the 2021 Splunk Career Survey.

Earn $50 in Amazon cash!