Why do I receive error "Search process did not exit cleanly, exit_code=255, description="exited with code 255"" and how to resolve it?

[Jamaal]

When I attempt to run a basic search, index=foo sourcetype=*, and any other search, I get the following error message:

[splunk-idx-e1] Search process did not exit cleanly, exit_code=255, description="exited with code 255". Please look in search.log for this peer in the Job Inspector for more info.
[splunk-idx-e2] Search process did not exit cleanly, exit_code=255, description="exited with code 255". Please look in search.log for this peer in the Job Inspector for more info.
[splunk-idx-w1] Search process did not exit cleanly, exit_code=255, description="exited with code 255". Please look in search.log for this peer in the Job Inspector for more info.
[splunk-idx-w2] Search process did not exit cleanly, exit_code=255, description="exited with code 255". Please look in search.log for this peer in the Job Inspector for more info.

My Splunk setup consist of a Master Node(Cluster Master), Search Head, and 4 Indexers.
When I click the inspect job and then click search.log I don't see any error messages. I was able to perform this search with no problems prior to today. Can anyone help with this issue? I can provide more information if needed.

[jcrabb_splunk]

Typically when we see this, its a problem with knowledge bundle replication from the SH to the Indexer. I'd make sure that you aren't out of disk space and that there isn't an app with a very long name causing a character limit issue. You can look at messages logged for "DistBundleRestHandler" and if you don't see anything obvious, set "DistBundleRestHandler" into DEBUG.

[Harishma]

Hi @jcrabb ,

Im facing the same issue. I get errors when I run the search from a particular app but the error isnt reported when I run the query from Search app.

•[ddd] Search process did not exit cleanly, exit_code=255, description="exited with code 255". Please look in search.log for this peer in the Job Inspector for more info.
•[ccc] Search process did not exit cleanly, exit_code=255, description="exited with code 255". Please look in search.log for this peer in the Job Inspector for more info.
•[bbb] Search process did not exit cleanly, exit_code=255, description="exited with code 255". Please look in search.log for this peer in the Job Inspector for more info.
•[aaa] Search process did not exit cleanly, exit_code=255, description="exited with code 255". Please look in search.log for this peer in the Job Inspector for more info.

Can you please let me know how to set "DistBundleRestHandler" into DEBUG. Is it just starting splunk in debug mode?

[jkat54]

Settings -> server settings -> logging