Search process did not exit cleanly, exit_code=255, description="exited with code 255". Please look in search.log for this peer in the Job Inspector for more info.
I've deployed a new app to our search head cluster and searches in this app are failing with above error.
The log files from all indexers says
ERROR dispatchRunner - RunDispatch::runDispatchThread threw error: Application does not exist: new_app
I've deployed the same app to indexer cluster
This search should give you more details
index=_internal source=*search.log log_level=WARN OR log_level=ERROR
index=_internal source=*searches.log log_level=WARN OR log_level=ERROR
I can search on command line of indexer
./bin/splunk search 'index=*|head 10' -app reports
How did you deploy the app to your search head cluster? It looks like it is not deployed as expected.
Also note that the
search.log for searches is NOT indexed by default, it is only available in the search inspector or in the dispatch directory until the search result expires.
I used the deployment server to deploy new app. extract the new app to etc/shcluster/apps/ and then apply shclusterbundle.
This command says it might restart the search heads, but it doesn't. So I've manually restarted one search head, launched new app and noticed this error.
1 - Agree with @MuS that there may be a problem with your app deployment
2 - I also wonder if all your search heads meet the minimum hardware/software requirements for Splunk?
Please provide all the .conf files in your new app. Complete but with sensitive data removed.
LOL @jkat54 - that sounds a Support ticket to me
I have no idea what else could be going wrong otherwise. I know the exit code 255 is from a subprocess routine in python (most likely), but I have no clue how to resolve without seeing everything in the new app.
Maybe you'd like to invite me to be on your team someday lquinn? I'd be happy to train folks on splunk and "troll" answers.splunk.com for pay. 😉
I do agree he/she should open a ticket at this point.