Scheduler stop working

[bckq]

I have about 150-200 scheduled searches that runs every minute. Most of searches look for data from 15 minutes till now. I noticed, that sometimes scheduler is stop working. Once it is for 2 minutes, sometimes even for 15 oraz 30 minutes. I monitor number of entries to scheduler.log and you can see the result of my tests. This situation affect on my dashboards. When scheduler has stopped, when I refresh my dashboard I will see old data, from last search done. This is serious problem, because I use Splunk for monitoring. This line should be always solid and constant.

Another thing is why Splunk runs only ~100 searches per minute if I have 200 to do.

I am using Splunk 5.0.4. One search head and two indexers.

[bckq]

IBM HS22 with 24 cores and 24GB RAM.

[_d_ · Accepted solution]

There is a finite number of searches that splunk will run concurrently and it depends on the resources (more specifically the number of CPU cores) that your system has. While you may want the scheduler to run 200 searches per minute it may not have the capacity to do so due to insufficient number of cores (or long runtimes of each search). When that capacity is reached, by default the scheduler will skip starting/executing of the next instance of a search unless you change governing defaults in limits.conf. However, the change will not really help in actually completing said search - it will only start it and the search will run/complete when the operating system has enough resources to do so (ie. when the currently running searches release them).

[_d_]

I cannot say why without looking at the scheduler.log or splunkd.log but I would assume that maybe it is because the run duration of those searches is such that the scheduler cannot run new searches until some finish.

[bckq]

I understand that I cannot run more searches in specified time, but why does the scheduler stop working for example 15-20 minutes and then start without reason again?

[alacercogitatus]

How much CPU/RAM does your Search Head / Indexers have?