Splunk Search

After deploying a new app to our search head cluster, why am I getting search error "Search process did not exit cleanly, exit_code=255"?

Communicator
Search process did not exit cleanly, exit_code=255, description="exited with code 255". Please look in search.log for this peer in the Job Inspector for more info.

I've deployed a new app to our search head cluster and searches in this app are failing with above error.
The log files from all indexers says

 ERROR dispatchRunner - RunDispatch::runDispatchThread threw error: Application does not exist: new_app

I've deployed the same app to indexer cluster

0 Karma
1 Solution

Communicator

I used the deployment server to deploy new app. extract the new app to etc/shcluster/apps/ and then apply shclusterbundle.
This command says it might restart the search heads, but it doesn't. So I've manually restarted one search head, launched new app and noticed this error.

Problem solved after restarting the other search heads. Before restarting the Captain, it (and other heads in cluster) doesn't know the new app, hence this error message. But i found this error message in Indexers log (inspect search->indexer log file) misleading me to deploy the app to indexer cluster before restarting all search heads.

I am not sure why this error appeared in search logs from indexers.

PS: I've noticed this on the same day, but could not post an update, sorry for the delay and thanks for the help

View solution in original post

Communicator

I used the deployment server to deploy new app. extract the new app to etc/shcluster/apps/ and then apply shclusterbundle.
This command says it might restart the search heads, but it doesn't. So I've manually restarted one search head, launched new app and noticed this error.

Problem solved after restarting the other search heads. Before restarting the Captain, it (and other heads in cluster) doesn't know the new app, hence this error message. But i found this error message in Indexers log (inspect search->indexer log file) misleading me to deploy the app to indexer cluster before restarting all search heads.

I am not sure why this error appeared in search logs from indexers.

PS: I've noticed this on the same day, but could not post an update, sorry for the delay and thanks for the help

View solution in original post

SplunkTrust
SplunkTrust

Please provide all the .conf files in your new app. Complete but with sensitive data removed.

0 Karma

Legend

LOL @jkat54 - that sounds a Support ticket to me

SplunkTrust
SplunkTrust

I have no idea what else could be going wrong otherwise. I know the exit code 255 is from a subprocess routine in python (most likely), but I have no clue how to resolve without seeing everything in the new app.

Maybe you'd like to invite me to be on your team someday lquinn? I'd be happy to train folks on splunk and "troll" answers.splunk.com for pay. 😉

I do agree he/she should open a ticket at this point.

0 Karma

Legend

1 - Agree with @MuS that there may be a problem with your app deployment

2 - I also wonder if all your search heads meet the minimum hardware/software requirements for Splunk?

SplunkTrust
SplunkTrust

This search should give you more details

index=_internal source=*search.log  log_level=WARN OR log_level=ERROR
0 Karma

Communicator

No results

index=_internal source=*searches.log log_level=WARN OR log_level=ERROR
0 Karma

Communicator

I can search on command line of indexer

./bin/splunk search 'index=*|head 10' -app reports
0 Karma

SplunkTrust
SplunkTrust

How did you deploy the app to your search head cluster? It looks like it is not deployed as expected.
Also note that the search.log for searches is NOT indexed by default, it is only available in the search inspector or in the dispatch directory until the search result expires.

cheers, MuS

Communicator

I used the deployment server to deploy new app. extract the new app to etc/shcluster/apps/ and then apply shclusterbundle.
This command says it might restart the search heads, but it doesn't. So I've manually restarted one search head, launched new app and noticed this error.

0 Karma