Hi,
In one of our indexer clusters which we query from a search head cluster, only one of the indexers is giving this error while running a search. The error I'm getting is:
<indexer_hostname>Search process did not exit cleanly, exit_code=255, description="exited with code 255". Please look in search.log for this peer in the Job Inspector for more info.
When going through search.log, for that particular indexer. All I can find is:
INFO DistributedSearchResultCollectionManager - Connecting to peer=<indexer> connectAll 0 connectToSpecificPeer 1
INFO DistributedSearchResultCollectionManager - Successfully created search result collector for peer=<indexer> in 0.002 seconds
And there aren't any ERROR in the search.log.
Although I found some errors in splunkd.log for the same indexer which is as below:
ERROR DistBundleRestHandler - Problem untarring file: /opt/splunk/var/run/searchpeers/xxx.bundle
WARN DistBundleRestHandler - There was a problem renaming: /opt/splunk/var/run/searchpeers/xxx.tmp -> /opt/splunk/var/run/searchpeers/xxxx: Directory not empty
I have seen some of the previous answers, stating that there might be not enough free space available in that particular indexer, but when checked, there is still 40% more available space.
I couldn't figure out what was the problem as there was no ERROR in the search.log. I'm on Splunk 7.1.3
Thanks in advance.
... View more