Splunk Enterprise Security

Unable to run searches in Splunk Enterprise Security because of the error "BUNDLE_SIZE_EXCEEDS_MAX_SIZE"

daniel_augustyn
Contributor

I am getting the following error in the Search Head running Splunk Enterprise Security:

Unable to distribute to peer named splunk_1 at uri https://x.x.x.x:8089 because replication was unsuccessful. replicationStatus Failed failure info: failed_because_BUNDLE_SIZE_EXCEEDS_MAX_SIZE Please verify connectivity to the search peer, that the search peer is up, and an adequate level of system resources are available. See the Troubleshooting Manual for more information.

I did some changes to distsearch.conf file, but the bundle is still over 3 GB in size.

This is the file stanza:

[replicationSettings]
maxBundleSize = 4096

[replicationSettings]
sendRcvTimeout = 1060

[replicationWhitelist]
allConf = *.conf
allSpec = *.spec

[replicationSettings:refineConf]
replicate.app               = false
replicate.authorize         = true
replicate.collections       = false
replicate.commands          = false
replicate.eventtypes        = false
replicate.fields            = false
replicate.segmenters        = false
replicate.literals          = false
replicate.lookups           = false
replicate.multikv           = false
replicate.props             = true
replicate.tags              = true
replicate.transforms        = true
replicate.transactiontypes  = false

[replicationBlacklist]
nopyc  = apps[/\\]...[/\\](bin|contrib|lib)[/\\]*.py[co]
nopyc2 = apps[/\\]...[/\\](bin|contrib|lib)[/\\]...[/\\]*.py[co]

nocsvdefault = apps[/\\]...[/\\]lookups[/\\]*.csv.default

nolearned = apps[/\\]learned[/\\]...

notracker  = apps[/\\]...[/\\]lookups[/\\]*tracker.csv
notracker2 = apps[/\\]...[/\\]lookups[/\\]*tracker2.csv
nosigref   = apps[/\\]...[/\\]lookups[/\\]*signature_reference.csv
nosigref2  = apps[/\\]...[/\\]lookups[/\\]*signature_reference2.csv


noeditablelookups = apps[/\\]...[/\\]lookups[/\\]editable_lookups.csv

nolegacycontexts = apps[/\\]...[/\\]contexts[/\\]*.context
noconvertedcontexts = apps[/\\]...[/\\]lookups[/\\]*.context.csv.old


noinstallsrc = apps[/\\]SplunkEnterpriseSecuritySuite[/\\]@installsrc@[/\\]...

lookupindexfiles = (system|apps/*|users(/_reserved)?/*/*)/lookups/*.(tmp$|index($|/...))
sampleapp = apps/sample_app/...

conf = (system|(apps/*))/(default|local)/server.conf

user_specific_meta = users(/_reserved)?/*/*/metadata/local.meta
framework = apps/framework/...
1 Solution

sjohnson_splunk
Splunk Employee
Splunk Employee

Go to the search head and look at the var/run/searchpeers directory. Copy one of the .bundle files to a temp directory and then untar it.

cd to the directory and run the command:

du -m --max-depth=1

This will show you the size of the different apps in the bundle. Find the largest one(s), cd into that and re-run the command. I suspect that you will find a very large lookup file(s). You will want to blacklist it in distsearch.conf on the search head.

View solution in original post

sjohnson_splunk
Splunk Employee
Splunk Employee

Go to the search head and look at the var/run/searchpeers directory. Copy one of the .bundle files to a temp directory and then untar it.

cd to the directory and run the command:

du -m --max-depth=1

This will show you the size of the different apps in the bundle. Find the largest one(s), cd into that and re-run the command. I suspect that you will find a very large lookup file(s). You will want to blacklist it in distsearch.conf on the search head.

Jeremiah
Motivator

Do you know what's taking up so much space in your bundle?

0 Karma

daniel_augustyn
Contributor

I really don't, how should I check that?

0 Karma

jkat54
SplunkTrust
SplunkTrust

http://docs.splunk.com/Documentation/Splunk/6.4.2/DistSearch/Whatsearchheadssend

On the search head, the knowledge bundles resides under the $SPLUNK_HOME/var/run directory.

The knowledge bundle gets distributed to the $SPLUNK_HOME/var/run/searchpeers directory on each search peer (indexer).

I'd start on the search heads with a simple du -sh /opt/splunk/var/run command (assuming your $SPLUNK_HOME is /opt/splunk

0 Karma

daniel_augustyn
Contributor

I did that and it shows few of 3GB files. The total is 35G of data in that folder. I already did that before but still don't know how to limit that.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...