Splunk Search

large delay between dispatch_time and scheduled_time in shc - SHCRepJob failed

schose
Contributor

Hi forum,

we are facing large increasing delays between dispatch_time and scheduled_time in scheduler log. We see delays up to 200 seconds for searches which are scheduled every minute - resulting in not-scheduled searches.

this directly correlates to "ERROR SHCRepJob - failed to delegate job job=SHPDelegateSearchJob peer="peername", guid="466C6F6D-2779-4BFD-AE1F-64A71D0A5AC8" saved_search=system;; err="
messages we see at the SHC Captain.

Any hints? Other Single-Search-head instances do not show any issues. Using v.7.1.6

Best Regards,

Andreas

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Hi,

Have you looked at scheduler.log on your Search Head Cluster Members ?

Below query will help you to identify why captain failed to delegate job to SHC members.

index=_internal host IN(SH1,SH2,SH3) sourcetype=scheduler status=delegated_remote_error
0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!