Can you please check in
$SPLUNK_HOME/var/log/splunk/scheduler.log ? You can try to run this query
index=_internal host=<SEARCHHEAD> source=*scheduler.log savedsearch_name=<SCHEDULED_SEARCHNAME> on search head and then you can check whether it triggered at scheduled time or not.
index=_internal source=*scheduler.log savedsearch_name=<report name>
It gives an error message at the scheduled time -
status=delegated_remote_error, error accessing https://<SH 2>:8089/servicesNS/<user id>/<app name>/shcluster/member/<report name>/sched_dispatch?output_mode=json, statusCode=404, description=Not Found
Which version of splunk are you running ? Because I have checked on 6.4.X and 6.6.X SH cluster and I can't find this URL
https://<SH >:8089/servicesNS/<user id>/<app name>/shcluster/member/<report name> , till member URL is exist but after that other options are available not
<report name> so I would suggest you to raise case with splunk.
I ran -
index=_internal sourcetype=*sched* source="*scheduler.log" | timechart span=1day count by status
It shows -
delegated_remote - 1782
delegated_remote_completion - 1781
delegated_remote_error - 415
We experiencing quite a few of status=delegated_remote_error and status=skipped searches on Splunk SHC ver 8.0.2
even with following setting:
Total Scheduled searches = (((base_max_searches + cpu_count*max_search_per_cpu) * max_searches_perc) / 100) * num_members
Total schedule searches = ((6 + 36*1) * 50/100) * 7 = 21 * 7 = 147
Is there other setting tuning variables?