Hi @ddrilic,
Can you please check in $SPLUNK_HOME/var/log/splunk/scheduler.log
? You can try to run this query index=_internal host=<SEARCHHEAD> source=*scheduler.log savedsearch_name=<SCHEDULED_SEARCHNAME>
on search head and then you can check whether it triggered at scheduled time or not.
Great @harsmarvania57.
I ran index=_internal source=*scheduler.log savedsearch_name=<report name>
It gives an error message at the scheduled time -
status=delegated_remote_error, error accessing https://<SH 2>:8089/servicesNS/<user id>/<app name>/shcluster/member/<report name>/sched_dispatch?output_mode=json, statusCode=404, description=Not Found
Which version of splunk are you running ? Because I have checked on 6.4.X and 6.6.X SH cluster and I can't find this URL https://<SH >:8089/servicesNS/<user id>/<app name>/shcluster/member/<report name>
, till member URL is exist but after that other options are available not <report name>
so I would suggest you to raise case with splunk.
Very interesting @harsmarvania57 - we are at 6.5.2. A case with Support was opened...
Based on Search Head Cluster scheduled searches: What are these Status values in scheduler.log?
I ran -
index=_internal sourcetype=*sched* source="*scheduler.log" | timechart span=1day count by status
It shows -
delegated_remote - 1782
delegated_remote_completion - 1781
delegated_remote_error - 415
We experiencing quite a few of status=delegated_remote_error and status=skipped searches on Splunk SHC ver 8.0.2
even with following setting:
Total Scheduled searches = (((base_max_searches + cpu_count*max_search_per_cpu) * max_searches_perc) / 100) * num_members
Total schedule searches = ((6 + 36*1) * 50/100) * 7 = 21 * 7 = 147
Is there other setting tuning variables?
Thanks.
Now was the alert created? On the search head (see host field) where you see status=delegated_remote_error, does it exists or have correct permissions?
On the search head that the error refers to, the alert exists. What should the permissions be?