failed_because_BUNDLE_DATA_TRANSMIT_FAILURE

[Venkataraman]

Dear Team,

We have a cluster step up where 3 search head(cluster) , 3 indexer(cluster),1 index master,1 deployer, 1 license master
We are getting below mentioned errors in the production system in one of the search head.

Error Message: "Unable to distribute to peer named xxx5012.xxxx.com at uri https://xx.41.xxx.xx:8089 because replication was unsuccessful. replicationStatus Failed failure info: failed_because_BUNDLE_DATA_TRANSMIT_FAILURE Please verify connectivity to the search peer, that the search peer is up, and an adequate level of system resources are available. See the Troubleshooting Manual for more information."

Can you please help us in solving it.

Regards,
Santosh

[m_pham]

This is an old post but I want to post resolutions that worked for us in case someone else runs into the same error.

You'll usually see these bundle replication errors with the search below (you'll need to edit the search with your search head and indexer hostnames - wildcard it if you want):

Note: the Monitoring Console app has a dashboard for these type of errors in Search > Knowledge Bundle Replication

index=_internal host IN (<YOUR_SH_HOSTNAME>, <YOUR_INDEXER_HOSTNAME>) source=*splunkd.log* (component=BundlesAdminHandler OR component=BundleDataProcessor OR component=BundleDeltaHandler OR component=BundleReplicationProvider OR component=BundleStatusManager OR component=BundleTransaction OR component=CascadePlan OR component=CascadeReplicationReaper OR component=CascadingBundleReplicationProvider OR component=CascadingReplicationManager OR component=CascadingReplicationTransaction OR component=CascadingReplicationStatusActor OR component=CascadingUploadHandler OR component=ClassicBundleReplicationProvider OR component=DistBundleRestHandler OR component=DistributedBundleReplicationManager OR component=GetCascadingReplicationStatusTransaction OR component=RFSManager OR component=RFSBundleReplicationProvider) (log_level=WARN OR log_level=ERROR) component=ClassicBundleReplicationProvider log_level=ERROR

In the error logs, note down the search head reporting the errors and the indexers listed in logs. Verify that the search head can connect to the indexers listed in the error log.

The second resolution is log into the search head that is reporting the error and check the timestamp of the content inside $SPLUNK_HOME/var/run/proxy_bundles (IE Linux command: ls -lah). If the timestamp of the files are more than a few days ago, then you would need to move the proxy_bundles directory to a backup location and restart Splunk; this should fix the errors.

[nawazns5038]

Did you check the bundle size that you are sending from the search heads to the indexers ?

[richgalloway]

Have followed the suggestions in the error message? Verified the SH can connect to the indexer? Checked the resources on the indexer? Read the Troubleshooting manual?

[Venkataraman]

Hi Richgalloway,

Thanks for replying.

We are looking into them already without any luck so far. Just to add to the above, we have recently upgraded our system.
SH's : c4.8x to c5.18x
Indexer: c4.8x to c5.9x
deployer:c4.8x to c5.9x
index master: c4.8x to c5.9x
license manager: c4.8x to c5.9x

Do we need to re authenticate search heads and indexers after upgrading?

Regards,
Santosh.

[richgalloway]

Someone else will have to answer that question.