topic Re: After deploying a new app to our search head cluster, why am I getting search error "Search process did not exit cleanly, exit_code=255"? in Splunk Search

After deploying a new app to our search head cluster, why am I getting search error "Search process did not exit cleanly, exit_code=255"?

i2sheri — Thu, 17 Dec 2015 02:34:06 GMT

Search process did not exit cleanly, exit_code=255, description="exited with code 255". Please look in search.log for this peer in the Job Inspector for more info.

I've deployed a new app to our search head cluster and searches in this app are failing with above error.
The log files from all indexers says

 ERROR dispatchRunner - RunDispatch::runDispatchThread threw error: Application does not exist: new_app

I've deployed the same app to indexer cluster

Re: After deploying a new app to our search head cluster, why am I getting search error "Search process did not exit cleanly, exit_code=255"?

jkat54 — Thu, 17 Dec 2015 02:38:41 GMT

This search should give you more details

index=_internal source=*search.log  log_level=WARN OR log_level=ERROR

Re: After deploying a new app to our search head cluster, why am I getting search error "Search process did not exit cleanly, exit_code=255"?

i2sheri — Thu, 17 Dec 2015 02:49:24 GMT

No results

index=_internal source=*searches.log log_level=WARN OR log_level=ERROR

Re: After deploying a new app to our search head cluster, why am I getting search error "Search process did not exit cleanly, exit_code=255"?

i2sheri — Thu, 17 Dec 2015 02:58:00 GMT

I can search on command line of indexer

./bin/splunk search 'index=*|head 10' -app reports

Re: After deploying a new app to our search head cluster, why am I getting search error "Search process did not exit cleanly, exit_code=255"?

MuS — Thu, 17 Dec 2015 03:21:43 GMT

How did you deploy the app to your search head cluster? It looks like it is not deployed as expected.
Also note that the search.log for searches is NOT indexed by default, it is only available in the search inspector or in the dispatch directory until the search result expires.

cheers, MuS

Re: After deploying a new app to our search head cluster, why am I getting search error "Search process did not exit cleanly, exit_code=255"?

lguinn2 — Thu, 17 Dec 2015 18:59:23 GMT

1 - Agree with @MuS that there may be a problem with your app deployment

2 - I also wonder if all your search heads meet the minimum hardware/software requirements for Splunk?

Re: After deploying a new app to our search head cluster, why am I getting search error "Search process did not exit cleanly, exit_code=255"?

jkat54 — Fri, 18 Dec 2015 12:06:42 GMT

Please provide all the .conf files in your new app. Complete but with sensitive data removed.

Re: After deploying a new app to our search head cluster, why am I getting search error "Search process did not exit cleanly, exit_code=255"?

lguinn2 — Fri, 18 Dec 2015 17:19:06 GMT

LOL @jkat54 - that sounds a Support ticket to me

Re: After deploying a new app to our search head cluster, why am I getting search error "Search process did not exit cleanly, exit_code=255"?

jkat54 — Sun, 20 Dec 2015 11:37:13 GMT

I have no idea what else could be going wrong otherwise. I know the exit code 255 is from a subprocess routine in python (most likely), but I have no clue how to resolve without seeing everything in the new app.

Maybe you'd like to invite me to be on your team someday lquinn? I'd be happy to train folks on splunk and "troll" answers.splunk.com for pay. 😉

I do agree he/she should open a ticket at this point.

Re: After deploying a new app to our search head cluster, why am I getting search error "Search process did not exit cleanly, exit_code=255"?

i2sheri — Mon, 21 Dec 2015 01:57:17 GMT

I used the deployment server to deploy new app. extract the new app to etc/shcluster/apps/ and then apply shclusterbundle.
This command says it might restart the search heads, but it doesn't. So I've manually restarted one search head, launched new app and noticed this error.

Re: After deploying a new app to our search head cluster, why am I getting search error "Search process did not exit cleanly, exit_code=255"?

i2sheri — Mon, 21 Dec 2015 02:01:15 GMT

Problem solved after restarting the other search heads. Before restarting the Captain, it (and other heads in cluster) doesn't know the new app, hence this error message. But i found this error message in Indexers log (inspect search->indexer log file) misleading me to deploy the app to indexer cluster before restarting all search heads.

I am not sure why this error appeared in search logs from indexers.

PS: I've noticed this on the same day, but could not post an update, sorry for the delay and thanks for the help