Splunk Enterprise Security setup: Why am I seeing error "unable to distribute to peer...because replication was unsuccessful."?

[daniel_augustyn]

I've been trying to set up the Splunk Enterprise Security app, but I came across an issue that I can't find references to online.

Unable to distribute to peer named indexer1_hostname at uri https://xx.xx.xx.xx:8089 because replication was unsuccessful. replicationStatus Failed failure info: failed_because_BUNDLE_SIZE_EXCEEDS_MAX_SIZE Please verify connectivity to the search peer, that the search peer is up, and an adequate level of system resources are available. See the Troubleshooting Manual for more information.

Can someone already came across a similar issue?

Thanks!

[Masa · Accepted solution]

Wish the message was more user-friendly. Splunk should improve this message.

The most important here is that why the bundle is so huge. Is it really needs to distribute to indexers? Often times, there are large files which are not required to send to indexers, and setting blacklist in distsearch.conf to reduce bundle size is the best practice. Otherwise, even if we tune configuration and allow sending a large bundle, there would be other issues such as timeout and/or performance issues.

Understanding the potential problem due to a large bundle size, this warning message mean the bundle was too big to send. Two factors/attributes contributes here.
1. maxBundleSize attribute of [replicationSettings] in distsearch.conf ( default is 1GB)
2. max_content_length attribute in [httpServer] in server.conf ( default is 800MB)

So, if you increase those two parameters to meet the size of your bundle, the message will be gone. However, most likely you start to see other issues by sending such large bundle; e.g. some timeout and slow performance, more memory usage etc.

[Masa]

1. Check bundle contents
   http://docs.splunk.com/Documentation/Splunk/6.4.2/DistSearch/Whatsearchheadssend
   => Latest bundle directory transferred from a search head will be in var/run/splunk/searchpeers// at an indexer
   => Or, you can use find bundle files in Search Head's var/run directory. It is a tar file. So, you can open a .bundle file by tar command.

2. How to add blacklist
   http://docs.splunk.com/Documentation/Splunk/6.4.2/DistSearch/Limittheknowledgebundlesize

For other errors, potentially indexer side issue. you might be able to find errors/warnings in indexer side.
In any case, I would recommend to reduce bundle size first.

[daniel_augustyn]

increasing both values seemed to fixed the issue. You also mentioned to check why the bundle is so huge. How would I do that? And finally, how would I blacklist in distseasrch.conf file.

Thanks!

[starcher]

It is also a good idea if you have large lookups you treat as scratch to blacklist from replication a naming scheme like temp_. Then name any lookups that are large and have no real need to go to the indexers start with that prefix. I run into this a lot when manipulating large asset tables.

[daniel_augustyn]

I found a pretty good read here http://docs.splunk.com/Documentation/Splunk/6.3.1/DistSearch/Limittheknowledgebundlesize