Need a little help here. I'm experiencing an error "Search process did not exit cleanly, exit_code=255, description="exited with code 255". Please look in search.log for this peer in the Job Inspector for more info.". I'm using Splunk 6.0.3, because of that error i cant search . I don't know the cause of this error. I also tried to look here in community if there are users out there experienced this, yes i found some but no concrete solution.
Please enlighten me 😞
Does the Job Inspector or search.log linked at the bottom of the Job Inspector have any further information?
Your search please? Did you check the search peers if they are up or not?
Is is a distributed search ? and do the remote search.log in the search inspector mentions errors about unknown users or roles ?
Because those are the classic symptoms when the searchbundles were not copied to the search-peers. (or were somehow expired).
A quick test, is to go to the mentioned search-peer and look for the search bundle folder.
$SPLUNK_HOME/var/run/searchpeer/ look at the modification time, and remove the bundle with the name of the search-head.
Then retry a search, you should see the new bundle be copied.
Check if any of your apps are blacklisted in distsearch.conf.
After the following line of error, you would find 1 more line, which will give you name of lookup or knowledge object which is failing while replication.
"Search process did not exit cleanly, exit_code=255",
...[server] Streamed search execute failed because: Error in 'lookup' command: The lookup table 'abc.csv' does not exist.
Try adding local=t in your search. This will direct Splunk to look for this csv only on search head and not indexer and remove the error.
Good Luck !!!