Splunk cron vs scheduler

[marceloalejandr]

Greeting Splunkers, 

I'm researching an issue with Splunk scheduled reports and I came across the .conf2017 material "Making the Most of the Splunk Scheduler" (see attached snippet of page 10 of the material).    The issue we're seeing is some scheduled jobs are not returning results but when the same SPL that's in the job is run real-time there are results.    The jobs are scheduled as a cron within the Splunk UI Schedule "Run on Cron Schedule".    I came cross the .conf2017 material and maybe found an issue or concern related to the issue.  

Can anyone please clarify a couple things: 

- the material mentions that Cron is "Limited to a single machine".   What does this mean and how does Splunk determine which machine/server to utilize?   

- we schedule most of jobs as Cron because it has a little more flexibility with the time to set the start time.  I also came across the limits.conf and authorized.conf documentation and found that all of the Splunk settings are still set to the default.   In further researching the issue, it seems there are approximately 30 jobs starting or running at 0400 when the job in question is not returning results.    So the other question is, are we hitting a system limit and can Splunk be optimized or tweaked to support more jobs and/or is the system limit causing the report to return no results?   If Splunk can be optimized/tweaked which parameters or settings needs to be changed?   

Any thoughts?   Thanks in advance for any help and insight.    Cheers!

[richgalloway]

I'm not sure anything on the left side of page 10 was ever true.  Jobs scheduled with cron syntax are subject to the quotas as other scheduled searches, can use schedule window and skew, and are scheduled across a SHC.

It's possible the job did not return results because it was skipped.  Splunk should have logged the skip - search for index=_internal sourcetype=Scheduler status=skipped.

Thirty jobs is too many to run at once on fewer than 60 SH CPUs. Re-distribute the search schedules for a more even run count.  The MC can help with that as can the dashboard at https://github.com/dpaper-splunk/public/blob/master/dashboards/extended_search_reporting.xml