Default LINE_BREAKER broken?

[cwilmoth]

We recently upgraded from 6.3.3 to 6.4.1 in an attempt to fix some performance issues. After upgrading, there were a ton of "Changing breaking behavior for event stream because MAX_EVENTS (256) was exceeded without a single event break..." for multiple data sources on our heavy forwarders. I struggled to figure out why and eventually just created a [default] stanza in the props.conf file that gets deployed to both of our heavy forwarders and put the default LINE_BREAKER = ([\r\n]+) in there. After deployment, events are breaking just fine (like they were before).

Is this a known issue? I did not see anything in the release notes.

Thanks.

[martin_mueller]

There's a second change, the without list has should linemerge set to true while the with list has it set to false. This tells Splunk to merge lines back together to whole events after applying the line breaker. Try setting should linemerge to false without setting the line breaker.

[jmallorquin]

Hi,

The problem is that you configured the F:\Splunk\etc\system\default\props.conf SHOULD_LINEMERGE in the default directory.
You should never change the configuration in this directory becouse when you upgrade splunk overwrite default files.

Hope i help you

[cwilmoth]

Nothing has been changed in the default directory. The props.conf file is dated 5/12/2016 just like all the other default files that were put in place by the 6.4.1 upgrade. The previous default files (6.3.3) were all dated 4/28/2015 and that old props.conf file also had SHOULD_LINEMERGE set to true.

From props.conf.spec:

SHOULD_LINEMERGE = [true|false]
* When set to true, Splunk combines several lines of data into a single
multiline event, based on the following configuration attributes.
* *Defaults to true.*

[martin_mueller]

Run $SPLUNK_HOME/bin/splunk cmd btool --debug props list that_sourcetype with and without the extra default stanza and compare the output.

[cwilmoth]

With:
F:\Splunk\etc\apps\Dso_deploy_hvy_fwdrs\default\props.conf [deepsecurity-system_events]
F:\Splunk\etc\system\default\props.conf ANNOTATE_PUNCT = True
F:\Splunk\etc\system\default\props.conf AUTO_KV_JSON = true
F:\Splunk\etc\system\default\props.conf BREAK_ONLY_BEFORE =
F:\Splunk\etc\system\default\props.conf BREAK_ONLY_BEFORE_DATE = True
F:\Splunk\etc\system\default\props.conf CHARSET = AUTO
F:\Splunk\etc\system\default\props.conf DATETIME_CONFIG = \etc\datetime.xml
F:\Splunk\etc\system\default\props.conf HEADER_MODE =
F:\Splunk\etc\system\default\props.conf LEARN_SOURCETYPE = true
F:\Splunk\etc\apps\Dso_deploy_hvy_fwdrs\default\props.conf LINE_BREAKER = ([\r\n]+)
F:\Splunk\etc\system\default\props.conf LINE_BREAKER_LOOKBEHIND = 100
F:\Splunk\etc\system\local\props.conf MAX_DAYS_AGO = 90
F:\Splunk\etc\system\default\props.conf MAX_DAYS_HENCE = 2
F:\Splunk\etc\system\default\props.conf MAX_DIFF_SECS_AGO = 3600
F:\Splunk\etc\system\default\props.conf MAX_DIFF_SECS_HENCE = 604800
F:\Splunk\etc\system\default\props.conf MAX_EVENTS = 256
F:\Splunk\etc\system\default\props.conf MAX_TIMESTAMP_LOOKAHEAD = 128
F:\Splunk\etc\system\default\props.conf MUST_BREAK_AFTER =
F:\Splunk\etc\system\default\props.conf MUST_NOT_BREAK_AFTER =
F:\Splunk\etc\system\default\props.conf MUST_NOT_BREAK_BEFORE =
F:\Splunk\etc\system\default\props.conf SEGMENTATION = indexing
F:\Splunk\etc\system\default\props.conf SEGMENTATION-all = full
F:\Splunk\etc\system\default\props.conf SEGMENTATION-inner = inner
F:\Splunk\etc\system\default\props.conf SEGMENTATION-outer = outer
F:\Splunk\etc\system\default\props.conf SEGMENTATION-raw = none
F:\Splunk\etc\system\default\props.conf SEGMENTATION-standard = standard
F:\Splunk\etc\apps\Dso_deploy_hvy_fwdrs\default\props.conf SHOULD_LINEMERGE = false
F:\Splunk\etc\system\default\props.conf TRANSFORMS =
F:\Splunk\etc\apps\rb_steelhead_ta\default\props.conf TRANSFORMS-riverbed_src = riverbed_src
F:\Splunk\etc\apps\Dso_deploy_hvy_fwdrs\default\props.conf TRANSFORMS-t3 = set-tm-fw-sourcetype,set-tm-log-sourcetype,set-tm-im-sourcetype,set-tm-ip-sourcetype,set-tm-ipsevents
F:\Splunk\etc\system\default\props.conf TRUNCATE = 10000
F:\Splunk\etc\system\default\props.conf detect_trailing_nulls = auto
F:\Splunk\etc\system\default\props.conf maxDist = 100
F:\Splunk\etc\system\default\props.conf priority =
F:\Splunk\etc\system\default\props.conf sourcetype =

Without:
F:\Splunk\etc\apps\Dso_deploy_hvy_fwdrs\default\props.conf [deepsecurity-system_events]
F:\Splunk\etc\system\default\props.conf ANNOTATE_PUNCT = True
F:\Splunk\etc\system\default\props.conf AUTO_KV_JSON = true
F:\Splunk\etc\system\default\props.conf BREAK_ONLY_BEFORE =
F:\Splunk\etc\system\default\props.conf BREAK_ONLY_BEFORE_DATE = True
F:\Splunk\etc\system\default\props.conf CHARSET = AUTO
F:\Splunk\etc\system\default\props.conf DATETIME_CONFIG = \etc\datetime.xml
F:\Splunk\etc\system\default\props.conf HEADER_MODE =
F:\Splunk\etc\system\default\props.conf LEARN_SOURCETYPE = true
F:\Splunk\etc\system\default\props.conf LINE_BREAKER_LOOKBEHIND = 100
F:\Splunk\etc\system\local\props.conf MAX_DAYS_AGO = 90
F:\Splunk\etc\system\default\props.conf MAX_DAYS_HENCE = 2
F:\Splunk\etc\system\default\props.conf MAX_DIFF_SECS_AGO = 3600
F:\Splunk\etc\system\default\props.conf MAX_DIFF_SECS_HENCE = 604800
F:\Splunk\etc\system\default\props.conf MAX_EVENTS = 256
F:\Splunk\etc\system\default\props.conf MAX_TIMESTAMP_LOOKAHEAD = 128
F:\Splunk\etc\system\default\props.conf MUST_BREAK_AFTER =
F:\Splunk\etc\system\default\props.conf MUST_NOT_BREAK_AFTER =
F:\Splunk\etc\system\default\props.conf MUST_NOT_BREAK_BEFORE =
F:\Splunk\etc\system\default\props.conf SEGMENTATION = indexing
F:\Splunk\etc\system\default\props.conf SEGMENTATION-all = full
F:\Splunk\etc\system\default\props.conf SEGMENTATION-inner = inner
F:\Splunk\etc\system\default\props.conf SEGMENTATION-outer = outer
F:\Splunk\etc\system\default\props.conf SEGMENTATION-raw = none
F:\Splunk\etc\system\default\props.conf SEGMENTATION-standard = standard
F:\Splunk\etc\system\default\props.conf SHOULD_LINEMERGE = True
F:\Splunk\etc\system\default\props.conf TRANSFORMS =
F:\Splunk\etc\apps\rb_steelhead_ta\default\props.conf TRANSFORMS-riverbed_src = riverbed_src
F:\Splunk\etc\apps\Dso_deploy_hvy_fwdrs\default\props.conf TRANSFORMS-t3 = set-tm-fw-sourcetype,set-tm-log-sourcetype,set-tm-im-sourcetype,set-tm-ip-sourcetype,set-tm-ipsevents
F:\Splunk\etc\system\default\props.conf TRUNCATE = 10000
F:\Splunk\etc\system\default\props.conf detect_trailing_nulls = auto
F:\Splunk\etc\system\default\props.conf maxDist = 100
F:\Splunk\etc\system\default\props.conf priority =
F:\Splunk\etc\system\default\props.conf sourcetype =