Why am I getting error "Search process did not exit cleanly, exit_code=255", causing searches to not run in Splunk 6.0.3?

[xisura]

Hi Ninjas!

Need a little help here. I'm experiencing an error "Search process did not exit cleanly, exit_code=255, description="exited with code 255". Please look in search.log for this peer in the Job Inspector for more info.". I'm using Splunk 6.0.3, because of that error i cant search . I don't know the cause of this error. I also tried to look here in community if there are users out there experienced this, yes i found some but no concrete solution.

Please enlighten me 😞

[kamal_jagga]

After the following line of error, you would find 1 more line, which will give you name of lookup or knowledge object which is failing while replication.

"Search process did not exit cleanly, exit_code=255",
...[server] Streamed search execute failed because: Error in 'lookup' command: The lookup table 'abc.csv' does not exist.

Try adding local=t in your search. This will direct Splunk to look for this csv only on search head and not indexer and remove the error.

Good Luck !!!

[yannK]

Is is a distributed search ? and do the remote search.log in the search inspector mentions errors about unknown users or roles ?
Because those are the classic symptoms when the searchbundles were not copied to the search-peers. (or were somehow expired).

A quick test, is to go to the mentioned search-peer and look for the search bundle folder.
$SPLUNK_HOME/var/run/searchpeer/ look at the modification time, and remove the bundle with the name of the search-head.
Then retry a search, you should see the new bundle be copied.

[ganji]

Check if any of your apps are blacklisted in distsearch.conf.

[linu1988]

Your search please? Did you check the search peers if they are up or not?

[martin_mueller]

Does the Job Inspector or search.log linked at the bottom of the Job Inspector have any further information?