Why is one of my indexers crashing frequently with error "replicationStatus Failed failure info: failed_because_BUNDLE_DATA_TRANSMIT_FAILURE"?

[dineshraj9]

One my Splunk indexers is crashing frequently and it is not returning results when searched. In the search head, I can see the error:

Unable to distribute to peer named server1 at uri https://server1:8089 because replication was unsuccessful. replicationStatus Failed failure info: failed_because_BUNDLE_DATA_TRANSMIT_FAILURE

Has anyone faced this issue?

[s2_splunk · Accepted solution]

Can you describe your setup in a bit more detail, please?
1. What version of Splunk?
2. What OS version?
3. How many indexers / search heads?
4. Clustered indexers (replication) or standalone/distributed?

On the surface, it appears your search head cannot communicate with the indexer (which is expected if your indexer is crashing). Have you taken a look at splunkd.log on the indexer to see whether any error messages are logged there?

[dineshraj9]

Splunk version 6.3.1 (build f3e41e4b37b2)

Setting -> Distributed Search -> Search peers, my server was showing as failed.

The issue was with the distributed search peer authentication failing for this particular indexer. After the credentials were updated, search heads are able to connect to the indexer and error is no longer showing up.

Thanks for the quick response!