Deployment Architecture

Distributed Search Replication Failure after 6.3 upgrade with error "replicationStatus Failed failure info: failed_because_NONE"

dolivasoh
Contributor

I've seen a few related issues on Answers, but not this specific error.

I have a deployment with a single search head, two indexers, and a cluster master. After upgrading to 6.3, my search head can no longer replicate the knowledge bundle to both indexers. Replication status says "Failed" in distributed search and when attempting a search, I see the following error for both indexers. Identifying info redacted.

Unable to distribute to peer named <indexer_name> at uri https://<indexer_ip>:8089 because replication was unsuccessful. replicationStatus Failed failure info: failed_because_NONE

Searches work just fine from my cluster master and replication says Successful there. Anyone know what's going on? I even started a completely fresh installation and rebuilt the cluster to no avail.

1 Solution

dolivasoh
Contributor

Found this lurking once I decided to pull from the cluster and search internal logs a bit further.

ERROR DistributedBundleReplicationManager - bundle size=1449MB, path=/opt/splunk/var/run/o-mgb-spsh001-1443883250.bundle, is too large for replication, max_size=1024MB. Check for any large unwanted files in $SPLUNK_HOME/etc/

I updated distsearch.conf to allow the very large bundle and things are running smoothly.

[distributedSearch]
disabled = 0
serverTimeout = 900
statusTimeout = 900

[replicationSettings]
replicationThreads = 8
maxBundleSize = 14438892420

[replicationBlacklist]
noBinDir = (.../bin/*)
nojavabin = apps/splunk_archiver/java-bin/...

View solution in original post

dolivasoh
Contributor

Found this lurking once I decided to pull from the cluster and search internal logs a bit further.

ERROR DistributedBundleReplicationManager - bundle size=1449MB, path=/opt/splunk/var/run/o-mgb-spsh001-1443883250.bundle, is too large for replication, max_size=1024MB. Check for any large unwanted files in $SPLUNK_HOME/etc/

I updated distsearch.conf to allow the very large bundle and things are running smoothly.

[distributedSearch]
disabled = 0
serverTimeout = 900
statusTimeout = 900

[replicationSettings]
replicationThreads = 8
maxBundleSize = 14438892420

[replicationBlacklist]
noBinDir = (.../bin/*)
nojavabin = apps/splunk_archiver/java-bin/...
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...