Why do we get the exited with code 255 errors?

[ddrillic]

When we search (on stage) using index = <index name>, no results come up and we get the following -

2 errors occurred while the search was executing. Therefore, search results might be incomplete. Hide errors.
    [<indexer1>] Search process did not exit cleanly, exit_code=255, description="exited with code 255". Please look in search.log for this peer in the Job Inspector for more info.
    [<indexer2>] Search process did not exit cleanly, exit_code=255, description="exited with code 255". Please look in search.log for this peer in the Job Inspector for more info.

What can it be?

[alacercogitatus]

I noticed you said (on stage). Does this mean you are searching production from staging? Do all the apps exist in both locations?

For example, you have a staging app called Stage_Search. You are in this app context, and try to do a distributed search to production. If Stage_Search does not exist as an app on the production indexers, it might throw a 255. Create the app on the indexers, doesn't need anything in it, just a stub is fine.

splunk create app Stage_Search

Do that on all the indexers (or use CM to push if in a cluster) and try your search again.

[ddrillic]

As far as I can recall, we ran out of disk space on the server.