Splunk Enterprise Security

Unable to run searches in Splunk Enterprise Security because of the error "BUNDLE_SIZE_EXCEEDS_MAX_SIZE"

Contributor

I am getting the following error in the Search Head running Splunk Enterprise Security:

Unable to distribute to peer named splunk_1 at uri https://x.x.x.x:8089 because replication was unsuccessful. replicationStatus Failed failure info: failed_because_BUNDLE_SIZE_EXCEEDS_MAX_SIZE Please verify connectivity to the search peer, that the search peer is up, and an adequate level of system resources are available. See the Troubleshooting Manual for more information.

I did some changes to distsearch.conf file, but the bundle is still over 3 GB in size.

This is the file stanza:

[replicationSettings]
maxBundleSize = 4096

[replicationSettings]
sendRcvTimeout = 1060

[replicationWhitelist]
allConf = *.conf
allSpec = *.spec

[replicationSettings:refineConf]
replicate.app               = false
replicate.authorize         = true
replicate.collections       = false
replicate.commands          = false
replicate.eventtypes        = false
replicate.fields            = false
replicate.segmenters        = false
replicate.literals          = false
replicate.lookups           = false
replicate.multikv           = false
replicate.props             = true
replicate.tags              = true
replicate.transforms        = true
replicate.transactiontypes  = false

[replicationBlacklist]
nopyc  = apps[/\\]...[/\\](bin|contrib|lib)[/\\]*.py[co]
nopyc2 = apps[/\\]...[/\\](bin|contrib|lib)[/\\]...[/\\]*.py[co]

nocsvdefault = apps[/\\]...[/\\]lookups[/\\]*.csv.default

nolearned = apps[/\\]learned[/\\]...

notracker  = apps[/\\]...[/\\]lookups[/\\]*tracker.csv
notracker2 = apps[/\\]...[/\\]lookups[/\\]*tracker2.csv
nosigref   = apps[/\\]...[/\\]lookups[/\\]*signature_reference.csv
nosigref2  = apps[/\\]...[/\\]lookups[/\\]*signature_reference2.csv


noeditablelookups = apps[/\\]...[/\\]lookups[/\\]editable_lookups.csv

nolegacycontexts = apps[/\\]...[/\\]contexts[/\\]*.context
noconvertedcontexts = apps[/\\]...[/\\]lookups[/\\]*.context.csv.old


noinstallsrc = apps[/\\]SplunkEnterpriseSecuritySuite[/\\]@installsrc@[/\\]...

lookupindexfiles = (system|apps/*|users(/_reserved)?/*/*)/lookups/*.(tmp$|index($|/...))
sampleapp = apps/sample_app/...

conf = (system|(apps/*))/(default|local)/server.conf

user_specific_meta = users(/_reserved)?/*/*/metadata/local.meta
framework = apps/framework/...
1 Solution

Splunk Employee
Splunk Employee

Go to the search head and look at the var/run/searchpeers directory. Copy one of the .bundle files to a temp directory and then untar it.

cd to the directory and run the command:

du -m --max-depth=1

This will show you the size of the different apps in the bundle. Find the largest one(s), cd into that and re-run the command. I suspect that you will find a very large lookup file(s). You will want to blacklist it in distsearch.conf on the search head.

View solution in original post

Splunk Employee
Splunk Employee

Go to the search head and look at the var/run/searchpeers directory. Copy one of the .bundle files to a temp directory and then untar it.

cd to the directory and run the command:

du -m --max-depth=1

This will show you the size of the different apps in the bundle. Find the largest one(s), cd into that and re-run the command. I suspect that you will find a very large lookup file(s). You will want to blacklist it in distsearch.conf on the search head.

View solution in original post

Motivator

Do you know what's taking up so much space in your bundle?

0 Karma

Contributor

I really don't, how should I check that?

0 Karma

SplunkTrust
SplunkTrust

http://docs.splunk.com/Documentation/Splunk/6.4.2/DistSearch/Whatsearchheadssend

On the search head, the knowledge bundles resides under the $SPLUNK_HOME/var/run directory.

The knowledge bundle gets distributed to the $SPLUNK_HOME/var/run/searchpeers directory on each search peer (indexer).

I'd start on the search heads with a simple du -sh /opt/splunk/var/run command (assuming your $SPLUNK_HOME is /opt/splunk

0 Karma

Contributor

I did that and it shows few of 3GB files. The total is 35G of data in that folder. I already did that before but still don't know how to limit that.

0 Karma