Deployment Architecture

failed_because_BUNDLE_DATA_TRANSMIT_FAILURE

Venkataraman
Engager

Dear Team,

We have a cluster step up where 3 search head(cluster) , 3 indexer(cluster),1 index master,1 deployer, 1 license master
We are getting below mentioned errors in the production system in one of the search head.

Error Message: "Unable to distribute to peer named xxx5012.xxxx.com at uri https://xx.41.xxx.xx:8089 because replication was unsuccessful. replicationStatus Failed failure info: failed_because_BUNDLE_DATA_TRANSMIT_FAILURE Please verify connectivity to the search peer, that the search peer is up, and an adequate level of system resources are available. See the Troubleshooting Manual for more information."

Can you please help us in solving it.

Regards,
Santosh

0 Karma

m_pham
Splunk Employee
Splunk Employee

This is an old post but I want to post resolutions that worked for us in case someone else runs into the same error.

You'll usually see these bundle replication errors with the search below (you'll need to edit the search with your search head and indexer hostnames - wildcard it if you want):

Note: the Monitoring Console app has a dashboard for these type of errors in Search > Knowledge Bundle Replication

 

index=_internal host IN (<YOUR_SH_HOSTNAME>, <YOUR_INDEXER_HOSTNAME>) source=*splunkd.log* (component=BundlesAdminHandler OR component=BundleDataProcessor OR component=BundleDeltaHandler OR component=BundleReplicationProvider OR component=BundleStatusManager OR component=BundleTransaction OR component=CascadePlan OR component=CascadeReplicationReaper OR component=CascadingBundleReplicationProvider OR component=CascadingReplicationManager OR component=CascadingReplicationTransaction OR component=CascadingReplicationStatusActor OR component=CascadingUploadHandler OR component=ClassicBundleReplicationProvider OR component=DistBundleRestHandler OR component=DistributedBundleReplicationManager OR component=GetCascadingReplicationStatusTransaction OR component=RFSManager OR component=RFSBundleReplicationProvider) (log_level=WARN OR log_level=ERROR) component=ClassicBundleReplicationProvider log_level=ERROR

 

In the error logs, note down the search head reporting the errors and the indexers listed in logs. Verify that the search head can connect to the indexers listed in the error log.

The second resolution is log into the search head that is reporting the error and check the timestamp of the content inside $SPLUNK_HOME/var/run/proxy_bundles (IE Linux command: ls -lah). If the timestamp of the files are more than a few days ago, then you would need to move the proxy_bundles directory to a backup location and restart Splunk; this should fix the errors.

nawazns5038
Builder

Did you check the bundle size that you are sending from the search heads to the indexers ?

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Have followed the suggestions in the error message? Verified the SH can connect to the indexer? Checked the resources on the indexer? Read the Troubleshooting manual?

---
If this reply helps you, Karma would be appreciated.

Venkataraman
Engager

Hi Richgalloway,

Thanks for replying.

We are looking into them already without any luck so far. Just to add to the above, we have recently upgraded our system.
SH's : c4.8x to c5.18x
Indexer: c4.8x to c5.9x
deployer:c4.8x to c5.9x
index master: c4.8x to c5.9x
license manager: c4.8x to c5.9x

Do we need to re authenticate search heads and indexers after upgrading?

Regards,
Santosh.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Someone else will have to answer that question.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...