Distributed Search Replication Failure after 6.3 upgrade with error "replicationStatus Failed failure info: failed_because_NONE"

[dolivasoh]

I've seen a few related issues on Answers, but not this specific error.

I have a deployment with a single search head, two indexers, and a cluster master. After upgrading to 6.3, my search head can no longer replicate the knowledge bundle to both indexers. Replication status says "Failed" in distributed search and when attempting a search, I see the following error for both indexers. Identifying info redacted.

Unable to distribute to peer named <indexer_name> at uri https://<indexer_ip>:8089 because replication was unsuccessful. replicationStatus Failed failure info: failed_because_NONE

Searches work just fine from my cluster master and replication says Successful there. Anyone know what's going on? I even started a completely fresh installation and rebuilt the cluster to no avail.

[dolivasoh · Accepted solution]

Found this lurking once I decided to pull from the cluster and search internal logs a bit further.

ERROR DistributedBundleReplicationManager - bundle size=1449MB, path=/opt/splunk/var/run/o-mgb-spsh001-1443883250.bundle, is too large for replication, max_size=1024MB. Check for any large unwanted files in $SPLUNK_HOME/etc/

I updated distsearch.conf to allow the very large bundle and things are running smoothly.

[distributedSearch]
disabled = 0
serverTimeout = 900
statusTimeout = 900

[replicationSettings]
replicationThreads = 8
maxBundleSize = 14438892420

[replicationBlacklist]
noBinDir = (.../bin/*)
nojavabin = apps/splunk_archiver/java-bin/...