Can not find all my realtime alerts in the Job manager with running status

[sudiptakp]

I have 30 realtime e-mail alerts configured in splunk.

In Splunk Manager, it shows that all these searches have been scheduled. I can see the timestamps for all these searches in the schedule column in manager.

However, when I visit the job manager window, I can not find all these 30 jobs with status=running. I can find only 17 of them and 3 others totaling to 21.

Following settings have been applied to limits.conf files found in /etc/system/local as well as /etc/system/default.

max_searches_per_cpu = 4

base_max_searches = 6

max_rt_search_multiplier = 3

My hardware has a CPU with 2 cores.

My splunk version = 5.0.2

[rsennett_splunk]

limits.conf as suggested by it's name is intended to set limits within the scope of your hardware. The rule of thumb, is that Splunk occupies 1 core per search whether it is real time or historical. Historical searches release the core when they finish, and a real-time search will release the core when the alert triggers (or never if you just keep it running) which means you have to have the core capacity to allow for that.

So the intention of limits.conf is for someone with say, 16 cpu's trying to reduce the load on all of them. It won't expand the use of your 2 cores however.

You haven't mentioned whether these real-time searches are looking at a window (30 seconds for instance) or All Real-Time. That will effect how the scheduler runs the jobs. It will attempt to schedule it to kick off, but if there is no cpu available at that time, it will reschedule itself for another window. This doc explains how the scheduler decides priority in relation to concurrency:
http://docs.splunk.com/Documentation/Splunk/5.0.2/Knowledge/Schedulesearchpriority

And this one explains the mechanics of a real-time search.
http://docs.splunk.com/Documentation/Splunk/latest/Search/Aboutrealtimesearches

Because you are scheduling real-time searches you'll want to take into consideration the information in both docs.

Meanwhile, to get your alerts working while you sort the rest out, consider whether all of those scheduled alerts are required to run against the data prior to indexing (that's what real-time is). It's possible that some of the alerts would still be valid if they ran say, every minute or even every 5 minutes on the already indexed data. The efficiency of that will make more sense after you glance at the doc.

Hope this helps!