Splunk Search
Highlighted

After deploying a new app to our search head cluster, why am I getting search error "Search process did not exit cleanly, exit_code=255"?

Communicator
Search process did not exit cleanly, exit_code=255, description="exited with code 255". Please look in search.log for this peer in the Job Inspector for more info.

I've deployed a new app to our search head cluster and searches in this app are failing with above error.
The log files from all indexers says

 ERROR dispatchRunner - RunDispatch::runDispatchThread threw error: Application does not exist: new_app

I've deployed the same app to indexer cluster

0 Karma
Highlighted

Re: After deploying a new app to our search head cluster, why am I getting search error "Search process did not exit cleanly, exit_code=255"?

SplunkTrust
SplunkTrust

This search should give you more details

index=_internal source=*search.log  log_level=WARN OR log_level=ERROR
0 Karma
Highlighted

Re: After deploying a new app to our search head cluster, why am I getting search error "Search process did not exit cleanly, exit_code=255"?

Communicator

No results

index=_internal source=*searches.log log_level=WARN OR log_level=ERROR
0 Karma
Highlighted

Re: After deploying a new app to our search head cluster, why am I getting search error "Search process did not exit cleanly, exit_code=255"?

Communicator

I can search on command line of indexer

./bin/splunk search 'index=*|head 10' -app reports
0 Karma
Highlighted

Re: After deploying a new app to our search head cluster, why am I getting search error "Search process did not exit cleanly, exit_code=255"?

SplunkTrust
SplunkTrust

How did you deploy the app to your search head cluster? It looks like it is not deployed as expected.
Also note that the search.log for searches is NOT indexed by default, it is only available in the search inspector or in the dispatch directory until the search result expires.

cheers, MuS

Highlighted

Re: After deploying a new app to our search head cluster, why am I getting search error "Search process did not exit cleanly, exit_code=255"?

Communicator

I used the deployment server to deploy new app. extract the new app to etc/shcluster/apps/ and then apply shclusterbundle.
This command says it might restart the search heads, but it doesn't. So I've manually restarted one search head, launched new app and noticed this error.

0 Karma
Highlighted

Re: After deploying a new app to our search head cluster, why am I getting search error "Search process did not exit cleanly, exit_code=255"?

Legend

1 - Agree with @MuS that there may be a problem with your app deployment

2 - I also wonder if all your search heads meet the minimum hardware/software requirements for Splunk?

Highlighted

Re: After deploying a new app to our search head cluster, why am I getting search error "Search process did not exit cleanly, exit_code=255"?

SplunkTrust
SplunkTrust

Please provide all the .conf files in your new app. Complete but with sensitive data removed.

0 Karma
Highlighted

Re: After deploying a new app to our search head cluster, why am I getting search error "Search process did not exit cleanly, exit_code=255"?

Legend

LOL @jkat54 - that sounds a Support ticket to me

Highlighted

Re: After deploying a new app to our search head cluster, why am I getting search error "Search process did not exit cleanly, exit_code=255"?

SplunkTrust
SplunkTrust

I have no idea what else could be going wrong otherwise. I know the exit code 255 is from a subprocess routine in python (most likely), but I have no clue how to resolve without seeing everything in the new app.

Maybe you'd like to invite me to be on your team someday lquinn? I'd be happy to train folks on splunk and "troll" answers.splunk.com for pay. 😉

I do agree he/she should open a ticket at this point.

0 Karma