Splunk Search

After deploying a new app to our search head cluster, why am I getting search error "Search process did not exit cleanly, exit_code=255"?

i2sheri
Communicator
Search process did not exit cleanly, exit_code=255, description="exited with code 255". Please look in search.log for this peer in the Job Inspector for more info.

I've deployed a new app to our search head cluster and searches in this app are failing with above error.
The log files from all indexers says

 ERROR dispatchRunner - RunDispatch::runDispatchThread threw error: Application does not exist: new_app

I've deployed the same app to indexer cluster

0 Karma
1 Solution

i2sheri
Communicator

I used the deployment server to deploy new app. extract the new app to etc/shcluster/apps/ and then apply shclusterbundle.
This command says it might restart the search heads, but it doesn't. So I've manually restarted one search head, launched new app and noticed this error.

Problem solved after restarting the other search heads. Before restarting the Captain, it (and other heads in cluster) doesn't know the new app, hence this error message. But i found this error message in Indexers log (inspect search->indexer log file) misleading me to deploy the app to indexer cluster before restarting all search heads.

I am not sure why this error appeared in search logs from indexers.

PS: I've noticed this on the same day, but could not post an update, sorry for the delay and thanks for the help

View solution in original post

i2sheri
Communicator

I used the deployment server to deploy new app. extract the new app to etc/shcluster/apps/ and then apply shclusterbundle.
This command says it might restart the search heads, but it doesn't. So I've manually restarted one search head, launched new app and noticed this error.

Problem solved after restarting the other search heads. Before restarting the Captain, it (and other heads in cluster) doesn't know the new app, hence this error message. But i found this error message in Indexers log (inspect search->indexer log file) misleading me to deploy the app to indexer cluster before restarting all search heads.

I am not sure why this error appeared in search logs from indexers.

PS: I've noticed this on the same day, but could not post an update, sorry for the delay and thanks for the help

jkat54
SplunkTrust
SplunkTrust

Please provide all the .conf files in your new app. Complete but with sensitive data removed.

0 Karma

lguinn2
Legend

LOL @jkat54 - that sounds a Support ticket to me

jkat54
SplunkTrust
SplunkTrust

I have no idea what else could be going wrong otherwise. I know the exit code 255 is from a subprocess routine in python (most likely), but I have no clue how to resolve without seeing everything in the new app.

Maybe you'd like to invite me to be on your team someday lquinn? I'd be happy to train folks on splunk and "troll" answers.splunk.com for pay. 😉

I do agree he/she should open a ticket at this point.

0 Karma

lguinn2
Legend

1 - Agree with @MuS that there may be a problem with your app deployment

2 - I also wonder if all your search heads meet the minimum hardware/software requirements for Splunk?

jkat54
SplunkTrust
SplunkTrust

This search should give you more details

index=_internal source=*search.log  log_level=WARN OR log_level=ERROR
0 Karma

i2sheri
Communicator

No results

index=_internal source=*searches.log log_level=WARN OR log_level=ERROR
0 Karma

i2sheri
Communicator

I can search on command line of indexer

./bin/splunk search 'index=*|head 10' -app reports
0 Karma

MuS
SplunkTrust
SplunkTrust

How did you deploy the app to your search head cluster? It looks like it is not deployed as expected.
Also note that the search.log for searches is NOT indexed by default, it is only available in the search inspector or in the dispatch directory until the search result expires.

cheers, MuS

i2sheri
Communicator

I used the deployment server to deploy new app. extract the new app to etc/shcluster/apps/ and then apply shclusterbundle.
This command says it might restart the search heads, but it doesn't. So I've manually restarted one search head, launched new app and noticed this error.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...