I created a few new scheduled searches through GUI. Their results are supposed to go to summary index.
I noticed they run only two three times after restarting Splunk, thne stop running. There is no "skipped" in scheduler.log. I cannot find the saved search name for the next schedule in audit.log, either.
When I run them manually, they always return results. It seems like the search itself does not matter. It's because this happens even the search is as simple as "index=* earliest=-5m@m | stats count".
What am I missing?
When a scheduled search for summary index is not run, and no logs, it's most likely your total number of the scheduled searches are more than the limit of Splunk.
Usually scheduled searches are skipped when it is not for summary index. This is usually acceptable. But, for summary index which might be used to generate for another report, skipping a scheduled search might result in incorrect report based on the summary index. So, Splunk try to run all the scheduled searches without skipping. Such type of scheduled search is called "continuous" scheduling in Splunk.
The attribute for this feature in savedsearches.conf is as follows;
realtime_schedule = 0
This means that the scheduled search should not skip and wait until it will run.
- That's the reason why the scheduled search never had "status=skipped" in the scheduler.log.
- When realtimeschedule is 0, it's called "continuous" scheduling.
- This "continuous" scheduling is default for a scheduled search to generate summary index
- This priority is lower than "realtimeschedule" search.
So, the "continuous" scheduled search is lower priority than general scheduled searches. It's because "continuous" scheduled search should be run whenever the CPU resource is available.
The problem is that when the general scheduled searches are always fully assigned to CPUs, the "continuous" scheduled searches have to wait for them to finish.
We have seen this issue when CPU is only two to four while the total number of scheduled searches are more than 150. But, again, it all depends on how long they need to run and how often they need to run.
You can enable DEBUG mode for scheduler.log and find these not-running "continuous" scheduled searches
What is the default of the maximum number of concurrent searches? It is defined by the following parameters in limits.conf files.
basemaxsearches (default is 4)
maxsearchespercpu (default is 4)
maxsearches_perc (default is 25, this means 25%)
Let's say your Splunk machine is 2 CPU core. So, numberOfCPU is 2
12(maxhistsearches) = 4(maxsearchpercpu) x 2(muberofcpus) + 4(basemax_searches)
You can run 12 concurrent searches manually.
For scheduled searches, it becomes only 3 ( = 12 x 25% ).
If this is the reason, and you know your CPU resource is not so busy all the time, you can try increasing value of the attributes. If this still not solve the issue, you should add another Splunk machine or increase the CPU cores.
( Please keep in mind that real time searches can run three times as many as the historical searches. )
Note that this info was correct before version v4.3.2.
Default values were changed in the current version. A scheduler.log show status=continued message if the summary index enabled scheduled search ( reatime_schedule = 0 ) was not run before the next schedule comes.