Why does my search produce error "Search process did not exit cleanly, exit_code=254" in my search head clustering environment?

[rbal_splunk]

What could cause this error? I have a distributed search environment with a Search Head having multiple indexers, and a Search Head Cluster with multiple indexers.

[rbal_splunk · Accepted solution]

If you see this often, then it should be investigated as well.
It might be something temporary or a lingering issue.

This message pops up when there is some issue with the search on one of the indexers.
More info should be in the remote_search.log for the indexer.

It can happen if the search dir was not found on the indexer, or
if there was some problem with bundle distribution and some knowledge objects are missing for the search on the indexer.

I also saw that when permissions were not the same on the SH and indexers, so the search on the indexer had some errors.