Deployment Architecture

Distributed Search Replication Failure after 6.3 upgrade with error "replicationStatus Failed failure info: failed_because_NONE"

dolivasoh
Contributor

I've seen a few related issues on Answers, but not this specific error.

I have a deployment with a single search head, two indexers, and a cluster master. After upgrading to 6.3, my search head can no longer replicate the knowledge bundle to both indexers. Replication status says "Failed" in distributed search and when attempting a search, I see the following error for both indexers. Identifying info redacted.

Unable to distribute to peer named <indexer_name> at uri https://<indexer_ip>:8089 because replication was unsuccessful. replicationStatus Failed failure info: failed_because_NONE

Searches work just fine from my cluster master and replication says Successful there. Anyone know what's going on? I even started a completely fresh installation and rebuilt the cluster to no avail.

1 Solution

dolivasoh
Contributor

Found this lurking once I decided to pull from the cluster and search internal logs a bit further.

ERROR DistributedBundleReplicationManager - bundle size=1449MB, path=/opt/splunk/var/run/o-mgb-spsh001-1443883250.bundle, is too large for replication, max_size=1024MB. Check for any large unwanted files in $SPLUNK_HOME/etc/

I updated distsearch.conf to allow the very large bundle and things are running smoothly.

[distributedSearch]
disabled = 0
serverTimeout = 900
statusTimeout = 900

[replicationSettings]
replicationThreads = 8
maxBundleSize = 14438892420

[replicationBlacklist]
noBinDir = (.../bin/*)
nojavabin = apps/splunk_archiver/java-bin/...

View solution in original post

dolivasoh
Contributor

Found this lurking once I decided to pull from the cluster and search internal logs a bit further.

ERROR DistributedBundleReplicationManager - bundle size=1449MB, path=/opt/splunk/var/run/o-mgb-spsh001-1443883250.bundle, is too large for replication, max_size=1024MB. Check for any large unwanted files in $SPLUNK_HOME/etc/

I updated distsearch.conf to allow the very large bundle and things are running smoothly.

[distributedSearch]
disabled = 0
serverTimeout = 900
statusTimeout = 900

[replicationSettings]
replicationThreads = 8
maxBundleSize = 14438892420

[replicationBlacklist]
noBinDir = (.../bin/*)
nojavabin = apps/splunk_archiver/java-bin/...
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...