Issue with ITSI service_health_monitor saved search

[Nisha18789]

Something weird started happening in our Splunk environment for ITSI native saved search : service_health_monitor

This search started getting 100% skipped with reason: The maximum number of concurrent running jobs for this historical scheduled search on this cluster has been reached

So, I checked the jobs section and found that the search was stuck running at x% < 100 and hence the next scheduled search could not start. So tried deleting that one, so that it can run in next run, but the next run showed the same behaviour ie, stuck halfway. 

Inspect job shows most of the time was spent on startup.handoff and below is what I can see in the end of savedsearch.log that after the noop process (BEGIN OPEN: Processor=noop) splunk seems stuck.

Please provide any insights which can help in investigating further.

09-07-2020 17:41:05.680 INFO  LocalCollector - Final required fields list = Message,_raw,_subsecond,_time,alert_level,alert_severity,app,index,indexed_is_service_max_severity_event,is_service_in_maintenance,itsi_kpi_id,itsi_service_id,kpi,kpiid,prestats_reserved_*,psrsvd_*,scoretype,service,serviceid,source,urgency
09-07-2020 17:41:05.680 INFO  UserManager - Unwound user context: splunk-system-user -> NULL
09-07-2020 17:41:05.680 INFO  UserManager - Setting user context: splunk-system-user
09-07-2020 17:41:05.680 INFO  UserManager - Done setting user context: NULL -> splunk-system-user
09-07-2020 17:41:05.680 INFO  UserManager - Unwound user context: splunk-system-user -> NULL
09-07-2020 17:41:06.105 INFO  UserManager - Unwound user context: splunk-system-user -> NULL
09-07-2020 17:41:06.105 INFO  UserManager - Unwound user context: splunk-system-user -> NULL
09-07-2020 17:41:06.105 INFO  UserManager - Unwound user context: splunk-system-user -> NULL
09-07-2020 17:41:06.105 INFO  UserManager - Unwound user context: splunk-system-user -> NULL
09-07-2020 17:41:06.105 INFO  UserManager - Unwound user context: splunk-system-user -> NULL
09-07-2020 17:41:06.171 INFO  ChunkedExternProcessor - Exiting custom search command after getinfo since we are in preview mode:gethealth
09-07-2020 17:41:06.177 INFO  SearchOrchestrator - Starting the status control thread.
09-07-2020 17:41:06.177 INFO  SearchOrchestrator - Starting phase=1
09-07-2020 17:41:06.177 INFO  UserManager - Setting user context: splunk-system-user
09-07-2020 17:41:06.177 INFO  UserManager - Setting user context: splunk-system-user
09-07-2020 17:41:06.177 INFO  UserManager - Done setting user context: NULL -> splunk-system-user
09-07-2020 17:41:06.177 INFO  UserManager - Done setting user context: NULL -> splunk-system-user
09-07-2020 17:41:06.177 INFO  ReducePhaseExecutor - Stating phase_1
09-07-2020 17:41:06.177 INFO  SearchStatusEnforcer - Enforcing disk quota = 26214400000
09-07-2020 17:41:06.177 INFO  PreviewExecutor - Preview Enforcing initialization done
09-07-2020 17:41:06.177 INFO  DispatchExecutor - BEGIN OPEN: Processor=stats
09-07-2020 17:41:06.209 INFO  ResultsCollationProcessor - Writing remote_event_providers.csv to disk
09-07-2020 17:41:06.209 INFO  DispatchExecutor - END OPEN: Processor=stats
09-07-2020 17:41:06.209 INFO  DispatchExecutor - BEGIN OPEN: Processor=gethealth
09-07-2020 17:41:06.217 INFO  DispatchExecutor - END OPEN: Processor=gethealth
09-07-2020 17:41:06.217 INFO  DispatchExecutor - BEGIN OPEN: Processor=noop
09-07-2020 17:48:07.948 INFO  ReducePhaseExecutor - ReducePhaseExecutor=1 action=PREVIEW