Unable to run searches in Splunk Enterprise Security because of the error "BUNDLE_SIZE_EXCEEDS_MAX_SIZE"

[daniel_augustyn]

I am getting the following error in the Search Head running Splunk Enterprise Security:

Unable to distribute to peer named splunk_1 at uri https://x.x.x.x:8089 because replication was unsuccessful. replicationStatus Failed failure info: failed_because_BUNDLE_SIZE_EXCEEDS_MAX_SIZE Please verify connectivity to the search peer, that the search peer is up, and an adequate level of system resources are available. See the Troubleshooting Manual for more information.

I did some changes to distsearch.conf file, but the bundle is still over 3 GB in size.

This is the file stanza:

[replicationSettings]
maxBundleSize = 4096

[replicationSettings]
sendRcvTimeout = 1060

[replicationWhitelist]
allConf = *.conf
allSpec = *.spec

[replicationSettings:refineConf]
replicate.app               = false
replicate.authorize         = true
replicate.collections       = false
replicate.commands          = false
replicate.eventtypes        = false
replicate.fields            = false
replicate.segmenters        = false
replicate.literals          = false
replicate.lookups           = false
replicate.multikv           = false
replicate.props             = true
replicate.tags              = true
replicate.transforms        = true
replicate.transactiontypes  = false

[replicationBlacklist]
nopyc  = apps[/\\]...[/\\](bin|contrib|lib)[/\\]*.py[co]
nopyc2 = apps[/\\]...[/\\](bin|contrib|lib)[/\\]...[/\\]*.py[co]

nocsvdefault = apps[/\\]...[/\\]lookups[/\\]*.csv.default

nolearned = apps[/\\]learned[/\\]...

notracker  = apps[/\\]...[/\\]lookups[/\\]*tracker.csv
notracker2 = apps[/\\]...[/\\]lookups[/\\]*tracker2.csv
nosigref   = apps[/\\]...[/\\]lookups[/\\]*signature_reference.csv
nosigref2  = apps[/\\]...[/\\]lookups[/\\]*signature_reference2.csv


noeditablelookups = apps[/\\]...[/\\]lookups[/\\]editable_lookups.csv

nolegacycontexts = apps[/\\]...[/\\]contexts[/\\]*.context
noconvertedcontexts = apps[/\\]...[/\\]lookups[/\\]*.context.csv.old


noinstallsrc = apps[/\\]SplunkEnterpriseSecuritySuite[/\\]@installsrc@[/\\]...

lookupindexfiles = (system|apps/*|users(/_reserved)?/*/*)/lookups/*.(tmp$|index($|/...))
sampleapp = apps/sample_app/...

conf = (system|(apps/*))/(default|local)/server.conf

user_specific_meta = users(/_reserved)?/*/*/metadata/local.meta
framework = apps/framework/...

[sjohnson_splunk · Accepted solution]

Go to the search head and look at the var/run/searchpeers directory. Copy one of the .bundle files to a temp directory and then untar it.

cd to the directory and run the command:

du -m --max-depth=1

This will show you the size of the different apps in the bundle. Find the largest one(s), cd into that and re-run the command. I suspect that you will find a very large lookup file(s). You will want to blacklist it in distsearch.conf on the search head.

[Jeremiah]

Do you know what's taking up so much space in your bundle?

[daniel_augustyn]

I really don't, how should I check that?

[jkat54]

http://docs.splunk.com/Documentation/Splunk/6.4.2/DistSearch/Whatsearchheadssend

On the search head, the knowledge bundles resides under the $SPLUNK_HOME/var/run directory.

The knowledge bundle gets distributed to the $SPLUNK_HOME/var/run/searchpeers directory on each search peer (indexer).

I'd start on the search heads with a simple du -sh /opt/splunk/var/run command (assuming your $SPLUNK_HOME is /opt/splunk

[daniel_augustyn]

I did that and it shows few of 3GB files. The total is 35G of data in that folder. I already did that before but still don't know how to limit that.