Search Returns Exit Code -2

[alacercogitatus]

Here's the situation. I have an international server. When trying to search it as a distributed peer, it exits with this message.

[REMOTE_WAN_HOST] Search process did not exit cleanly, exit_code=-2, description="exited with code -2". Please look in search.log for this peer in the Job Inspector for more info.

So I take a look in the peer's dispatch directory for the search log. I find this.


08-27-2013 00:56:51.884 INFO UserManager - Setting user context: USER
08-27-2013 00:56:51.884 WARN AuthorizationManager - Unknown role 'admin'
08-27-2013 00:56:51.884 WARN AuthorizationManager - Unknown role 'ess_admin'
08-27-2013 00:56:51.884 WARN AuthorizationManager - Unknown role 'ess_analyst'
08-27-2013 00:56:51.884 WARN AuthorizationManager - Unknown role 'ess_user'
08-27-2013 00:56:51.884 WARN AuthorizationManager - Unknown role 'power'
08-27-2013 00:56:51.884 WARN AuthorizationManager - Unknown role 'user'
08-27-2013 00:56:51.884 ERROR UserManagerPro - user="USER" had no roles
08-27-2013 00:56:51.884 ERROR UserManager - Error while setting user context: user="USER" had no roles
08-27-2013 00:56:51.884 INFO UserManager - Done setting user context: NULL -> NULL
08-27-2013 00:56:51.884 INFO UserManager - Unwound user context: NULL -> NULL
08-27-2013 00:56:51.884 ERROR dispatchRunner - RunDispatch::runDispatchThread threw error: Application does not exist: search


All of these roles and the USER exist in exactly the same way as on the search head. I'm at a loss on why it's not working correctly.

[brod_geico]

check this url, i had similar issues then i found version mismatch on SH peers.

http://answers.splunk.com/answers/192592/search-process-did-not-exit-cleanly-exit-code255-d.html#ans...

[brod_geico]

Few places you can check.
Search peers are running with inconsistency version.
Check search peers status .
ask user and that job/lookup table not had right permissions or not exist all over the places.

[David]

I had this same error... I tried a number of troubleshooting steps, but by the time I upped the debug level it randomly started working again (after failing for more than a day). I assume that timing is just coincidental. An upgrade to 6.0.1 didn't resolve it, and restarting the entire environment didn't resolve it. Also deleting and re-adding the search peer didn't resolve it. After the last restart, it still failed.. and then randomly begun working again.

[aelliott]

could this be the same issue? http://answers.splunk.com/answers/84199/distributed-search-application-does-not-exist-search

[somesoni2]

Hey..I am facing similar issue, were you able to resolved it?