topic Unable to run searches in Splunk Enterprise Security because of the error "BUNDLE_SIZE_EXCEEDS_MAX_SIZE" in Splunk Enterprise Security

Unable to run searches in Splunk Enterprise Security because of the error "BUNDLE_SIZE_EXCEEDS_MAX_SIZE"

daniel_augustyn — Tue, 09 Aug 2016 22:01:39 GMT

I am getting the following error in the Search Head running Splunk Enterprise Security:

Unable to distribute to peer named splunk_1 at uri https://x.x.x.x:8089 because replication was unsuccessful. replicationStatus Failed failure info: failed_because_BUNDLE_SIZE_EXCEEDS_MAX_SIZE Please verify connectivity to the search peer, that the search peer is up, and an adequate level of system resources are available. See the Troubleshooting Manual for more information.

I did some changes to distsearch.conf file, but the bundle is still over 3 GB in size.

This is the file stanza:

[replicationSettings]
maxBundleSize = 4096

[replicationSettings]
sendRcvTimeout = 1060

[replicationWhitelist]
allConf = *.conf
allSpec = *.spec

[replicationSettings:refineConf]
replicate.app               = false
replicate.authorize         = true
replicate.collections       = false
replicate.commands          = false
replicate.eventtypes        = false
replicate.fields            = false
replicate.segmenters        = false
replicate.literals          = false
replicate.lookups           = false
replicate.multikv           = false
replicate.props             = true
replicate.tags              = true
replicate.transforms        = true
replicate.transactiontypes  = false

[replicationBlacklist]
nopyc  = apps[/\\]...[/\\](bin|contrib|lib)[/\\]*.py[co]
nopyc2 = apps[/\\]...[/\\](bin|contrib|lib)[/\\]...[/\\]*.py[co]

nocsvdefault = apps[/\\]...[/\\]lookups[/\\]*.csv.default

nolearned = apps[/\\]learned[/\\]...

notracker  = apps[/\\]...[/\\]lookups[/\\]*tracker.csv
notracker2 = apps[/\\]...[/\\]lookups[/\\]*tracker2.csv
nosigref   = apps[/\\]...[/\\]lookups[/\\]*signature_reference.csv
nosigref2  = apps[/\\]...[/\\]lookups[/\\]*signature_reference2.csv


noeditablelookups = apps[/\\]...[/\\]lookups[/\\]editable_lookups.csv

nolegacycontexts = apps[/\\]...[/\\]contexts[/\\]*.context
noconvertedcontexts = apps[/\\]...[/\\]lookups[/\\]*.context.csv.old


noinstallsrc = apps[/\\]SplunkEnterpriseSecuritySuite[/\\]@installsrc@[/\\]...

lookupindexfiles = (system|apps/*|users(/_reserved)?/*/*)/lookups/*.(tmp$|index($|/...))
sampleapp = apps/sample_app/...

conf = (system|(apps/*))/(default|local)/server.conf

user_specific_meta = users(/_reserved)?/*/*/metadata/local.meta
framework = apps/framework/...

Re: Unable to run searches in Splunk Enterprise Security because of the error "BUNDLE_SIZE_EXCEEDS_MAX_SIZE"

Jeremiah — Wed, 10 Aug 2016 02:19:33 GMT

Do you know what's taking up so much space in your bundle?

Re: Unable to run searches in Splunk Enterprise Security because of the error "BUNDLE_SIZE_EXCEEDS_MAX_SIZE"

daniel_augustyn — Wed, 10 Aug 2016 16:56:54 GMT

I really don't, how should I check that?

Re: Unable to run searches in Splunk Enterprise Security because of the error "BUNDLE_SIZE_EXCEEDS_MAX_SIZE"

jkat54 — Wed, 10 Aug 2016 17:10:45 GMT

http://docs.splunk.com/Documentation/Splunk/6.4.2/DistSearch/Whatsearchheadssend

On the search head, the knowledge bundles resides under the $SPLUNK_HOME/var/run directory.

The knowledge bundle gets distributed to the $SPLUNK_HOME/var/run/searchpeers directory on each search peer (indexer).

I'd start on the search heads with a simple du -sh /opt/splunk/var/run command (assuming your $SPLUNK_HOME is /opt/splunk

Re: Unable to run searches in Splunk Enterprise Security because of the error "BUNDLE_SIZE_EXCEEDS_MAX_SIZE"

daniel_augustyn — Wed, 10 Aug 2016 19:25:42 GMT

I did that and it shows few of 3GB files. The total is 35G of data in that folder. I already did that before but still don't know how to limit that.

Re: Unable to run searches in Splunk Enterprise Security because of the error "BUNDLE_SIZE_EXCEEDS_MAX_SIZE"

sjohnson_splunk — Wed, 10 Aug 2016 19:38:54 GMT

Go to the search head and look at the var/run/searchpeers directory. Copy one of the .bundle files to a temp directory and then untar it.

cd to the directory and run the command:

du -m --max-depth=1

This will show you the size of the different apps in the bundle. Find the largest one(s), cd into that and re-run the command. I suspect that you will find a very large lookup file(s). You will want to blacklist it in distsearch.conf on the search head.