Deployment Architecture

failed_because_BUNDLE_DATA_TRANSMIT_FAILURE

Venkataraman
Engager

Dear Team,

We have a cluster step up where 3 search head(cluster) , 3 indexer(cluster),1 index master,1 deployer, 1 license master
We are getting below mentioned errors in the production system in one of the search head.

Error Message: "Unable to distribute to peer named xxx5012.xxxx.com at uri https://xx.41.xxx.xx:8089 because replication was unsuccessful. replicationStatus Failed failure info: failed_because_BUNDLE_DATA_TRANSMIT_FAILURE Please verify connectivity to the search peer, that the search peer is up, and an adequate level of system resources are available. See the Troubleshooting Manual for more information."

Can you please help us in solving it.

Regards,
Santosh

0 Karma

m_pham
Splunk Employee
Splunk Employee

This is an old post but I want to post resolutions that worked for us in case someone else runs into the same error.

You'll usually see these bundle replication errors with the search below (you'll need to edit the search with your search head and indexer hostnames - wildcard it if you want):

Note: the Monitoring Console app has a dashboard for these type of errors in Search > Knowledge Bundle Replication

 

index=_internal host IN (<YOUR_SH_HOSTNAME>, <YOUR_INDEXER_HOSTNAME>) source=*splunkd.log* (component=BundlesAdminHandler OR component=BundleDataProcessor OR component=BundleDeltaHandler OR component=BundleReplicationProvider OR component=BundleStatusManager OR component=BundleTransaction OR component=CascadePlan OR component=CascadeReplicationReaper OR component=CascadingBundleReplicationProvider OR component=CascadingReplicationManager OR component=CascadingReplicationTransaction OR component=CascadingReplicationStatusActor OR component=CascadingUploadHandler OR component=ClassicBundleReplicationProvider OR component=DistBundleRestHandler OR component=DistributedBundleReplicationManager OR component=GetCascadingReplicationStatusTransaction OR component=RFSManager OR component=RFSBundleReplicationProvider) (log_level=WARN OR log_level=ERROR) component=ClassicBundleReplicationProvider log_level=ERROR

 

In the error logs, note down the search head reporting the errors and the indexers listed in logs. Verify that the search head can connect to the indexers listed in the error log.

The second resolution is log into the search head that is reporting the error and check the timestamp of the content inside $SPLUNK_HOME/var/run/proxy_bundles (IE Linux command: ls -lah). If the timestamp of the files are more than a few days ago, then you would need to move the proxy_bundles directory to a backup location and restart Splunk; this should fix the errors.

nawazns5038
Builder

Did you check the bundle size that you are sending from the search heads to the indexers ?

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Have followed the suggestions in the error message? Verified the SH can connect to the indexer? Checked the resources on the indexer? Read the Troubleshooting manual?

---
If this reply helps you, Karma would be appreciated.

Venkataraman
Engager

Hi Richgalloway,

Thanks for replying.

We are looking into them already without any luck so far. Just to add to the above, we have recently upgraded our system.
SH's : c4.8x to c5.18x
Indexer: c4.8x to c5.9x
deployer:c4.8x to c5.9x
index master: c4.8x to c5.9x
license manager: c4.8x to c5.9x

Do we need to re authenticate search heads and indexers after upgrading?

Regards,
Santosh.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Someone else will have to answer that question.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...