Splunk Enterprise Security

Unable to run searches in Splunk Enterprise Security because of the error "BUNDLE_SIZE_EXCEEDS_MAX_SIZE"

daniel_augustyn
Contributor

I am getting the following error in the Search Head running Splunk Enterprise Security:

Unable to distribute to peer named splunk_1 at uri https://x.x.x.x:8089 because replication was unsuccessful. replicationStatus Failed failure info: failed_because_BUNDLE_SIZE_EXCEEDS_MAX_SIZE Please verify connectivity to the search peer, that the search peer is up, and an adequate level of system resources are available. See the Troubleshooting Manual for more information.

I did some changes to distsearch.conf file, but the bundle is still over 3 GB in size.

This is the file stanza:

[replicationSettings]
maxBundleSize = 4096

[replicationSettings]
sendRcvTimeout = 1060

[replicationWhitelist]
allConf = *.conf
allSpec = *.spec

[replicationSettings:refineConf]
replicate.app               = false
replicate.authorize         = true
replicate.collections       = false
replicate.commands          = false
replicate.eventtypes        = false
replicate.fields            = false
replicate.segmenters        = false
replicate.literals          = false
replicate.lookups           = false
replicate.multikv           = false
replicate.props             = true
replicate.tags              = true
replicate.transforms        = true
replicate.transactiontypes  = false

[replicationBlacklist]
nopyc  = apps[/\\]...[/\\](bin|contrib|lib)[/\\]*.py[co]
nopyc2 = apps[/\\]...[/\\](bin|contrib|lib)[/\\]...[/\\]*.py[co]

nocsvdefault = apps[/\\]...[/\\]lookups[/\\]*.csv.default

nolearned = apps[/\\]learned[/\\]...

notracker  = apps[/\\]...[/\\]lookups[/\\]*tracker.csv
notracker2 = apps[/\\]...[/\\]lookups[/\\]*tracker2.csv
nosigref   = apps[/\\]...[/\\]lookups[/\\]*signature_reference.csv
nosigref2  = apps[/\\]...[/\\]lookups[/\\]*signature_reference2.csv


noeditablelookups = apps[/\\]...[/\\]lookups[/\\]editable_lookups.csv

nolegacycontexts = apps[/\\]...[/\\]contexts[/\\]*.context
noconvertedcontexts = apps[/\\]...[/\\]lookups[/\\]*.context.csv.old


noinstallsrc = apps[/\\]SplunkEnterpriseSecuritySuite[/\\]@installsrc@[/\\]...

lookupindexfiles = (system|apps/*|users(/_reserved)?/*/*)/lookups/*.(tmp$|index($|/...))
sampleapp = apps/sample_app/...

conf = (system|(apps/*))/(default|local)/server.conf

user_specific_meta = users(/_reserved)?/*/*/metadata/local.meta
framework = apps/framework/...
1 Solution

sjohnson_splunk
Splunk Employee
Splunk Employee

Go to the search head and look at the var/run/searchpeers directory. Copy one of the .bundle files to a temp directory and then untar it.

cd to the directory and run the command:

du -m --max-depth=1

This will show you the size of the different apps in the bundle. Find the largest one(s), cd into that and re-run the command. I suspect that you will find a very large lookup file(s). You will want to blacklist it in distsearch.conf on the search head.

View solution in original post

sjohnson_splunk
Splunk Employee
Splunk Employee

Go to the search head and look at the var/run/searchpeers directory. Copy one of the .bundle files to a temp directory and then untar it.

cd to the directory and run the command:

du -m --max-depth=1

This will show you the size of the different apps in the bundle. Find the largest one(s), cd into that and re-run the command. I suspect that you will find a very large lookup file(s). You will want to blacklist it in distsearch.conf on the search head.

Jeremiah
Motivator

Do you know what's taking up so much space in your bundle?

0 Karma

daniel_augustyn
Contributor

I really don't, how should I check that?

0 Karma

jkat54
SplunkTrust
SplunkTrust

http://docs.splunk.com/Documentation/Splunk/6.4.2/DistSearch/Whatsearchheadssend

On the search head, the knowledge bundles resides under the $SPLUNK_HOME/var/run directory.

The knowledge bundle gets distributed to the $SPLUNK_HOME/var/run/searchpeers directory on each search peer (indexer).

I'd start on the search heads with a simple du -sh /opt/splunk/var/run command (assuming your $SPLUNK_HOME is /opt/splunk

0 Karma

daniel_augustyn
Contributor

I did that and it shows few of 3GB files. The total is 35G of data in that folder. I already did that before but still don't know how to limit that.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...