Why our hourly report doesn't show up?

[ddrillic]

We have an hourly report that runs manually just fine but the scheduled run doesn't seem to work. Where do we look?

Some reports of this user work and some don't.

[harsmarvania57]

Hi @ddrilic,

Can you please check in $SPLUNK_HOME/var/log/splunk/scheduler.log ? You can try to run this query index=_internal host=<SEARCHHEAD> source=*scheduler.log savedsearch_name=<SCHEDULED_SEARCHNAME> on search head and then you can check whether it triggered at scheduled time or not.

[ddrillic]

Great @harsmarvania57.

I ran index=_internal source=*scheduler.log savedsearch_name=<report name>

It gives an error message at the scheduled time -

status=delegated_remote_error, error accessing https://<SH 2>:8089/servicesNS/<user id>/<app name>/shcluster/member/<report name>/sched_dispatch?output_mode=json, statusCode=404, description=Not Found

[harsmarvania57]

Which version of splunk are you running ? Because I have checked on 6.4.X and 6.6.X SH cluster and I can't find this URL https://<SH >:8089/servicesNS/<user id>/<app name>/shcluster/member/<report name> , till member URL is exist but after that other options are available not <report name> so I would suggest you to raise case with splunk.

[ddrillic]

Very interesting @harsmarvania57 - we are at 6.5.2. A case with Support was opened...

[ddrillic]

Based on Search Head Cluster scheduled searches: What are these Status values in scheduler.log?

I ran -

index=_internal sourcetype=*sched* source="*scheduler.log" | timechart span=1day count by status

It shows -
delegated_remote - 1782
delegated_remote_completion - 1781
delegated_remote_error - 415

[lim2]

We experiencing quite a few of status=delegated_remote_error and status=skipped searches on Splunk SHC ver 8.0.2

even with following setting:

Total Scheduled searches = (((base_max_searches + cpu_count*max_search_per_cpu) * max_searches_perc) / 100) * num_members
Total schedule searches = ((6 + 36*1) * 50/100) * 7 = 21 * 7 = 147

Is there other setting tuning variables?

Thanks.

[somesoni2]

Now was the alert created? On the search head (see host field) where you see status=delegated_remote_error, does it exists or have correct permissions?

[ddrillic]

On the search head that the error refers to, the alert exists. What should the permissions be?