Why am I seeing "Unable to distribute to peer...because replication was unsuccessful" on an indexer that is not part of an indexer cluster?

[jitsinha]

For last couple of days I have been receiving following message from Splunk indexer

Unable to distribute to peer named DEV_IDX_01 at uri https://XXXXXXXX.XXXXXXXX.com:8089 because replication was unsuccessful. replicationStatus Failed

Can someone please ut some light from this front??

FYI this indexer is not part of any Index cluster

[yannK]

The replication in the message is the "search knowledge bundle replication".
Do not get confused with the indexing cluster replication or the search-head clustering replication.

This is the step before the search, when the search-head synchronize the bundle of all the apps and profiles to send to the indexers to run the search with the same context.

see http://docs.splunk.com/Documentation/Splunk/6.2.2/DistSearch/Whatsearchheadssend

Check for errors about bundle size, maybe is it too large. -> you can look at the timeouts in distsearch.conf to increase them
Maybe the indexer is not responsive, or slow

Also look on the indexers on the $SPLUNK_HOME/var/run/searchpeer folder, look if you see recent bundle (and untar bundle). They are at least one per search-head. Check permissions, and if needed move the files/folders aside, and retry to search, a new one should be resent.

[jitsinha]

Thnkx yannk. FYI I am using Splunk 5.0.5

[esix_splunk]

There are a few components that could be causing this. Are your searches to this indexer failing? This could be from search bundles not being distributed because of out of disk space on the indexer..

[jitsinha]

Yes the searches are failing.
After getting this error for few minutes I don't see any data for any search queries. But after couple of minutes if I search again it populates data based on my search query from search head end.
I checked and I have 53% space left in my indexer.

[masonmorales]

You might want to contact Splunk Support. They will ask you to run ./splunk diag on the indexers in question.