Real time searches are not running, and searching for one of the saved search names in the _internal index shows:
status=skipped, reason="Real time searches pending"
What does "Real time searches pending mean"? What are possible causes of this?
This is on Splunk Enterprise 6.4.1 and I do not see anything in the _internal index relating to the concurrent search limit being met.
Who's running those realtime searches? Does the user/role has the ability to run realtime searches? If not, you will see an error stating "Skipping searches as xyz does not have permissions etc". Take a look at the scheduler.log and find out.
Other reason would be, if there's not enough horse power on the search heads.
Hope this helps!
Each realtime search unpreemptively locks 1 core on EVERY INDEXER and on your Search Head. If you have more realtime searches than cores, you will get this error. If you are running this many realtime searches, you are looking for trouble and can almost surely be doing something nicer/better.
Here's my question: What is the best way to convert a real-time alert to something more manageable, and yet maintain a similar net effect?
Schedule it to cover a span of
X and run it every
X/2. This covers the case where events at the end of span
t an the beginning of
t+1 would just miss triggering in those windows but will hit in the next alert run. Then make
X as large as you can stomach.
Do you suggest doing the cron scheduling format? One of these is a situation where we're monitoring account lockouts, so our need is for it to be accurate enough to act upon -- say 5 minutes. What about duplicates?
The only difference in the cron format is that it gives you complete granularity.
Cron for every 5 minutes is
*/5 * * * *. Alert once per user and throttle accordingly with the throttling settings.
Unless I'm missing something, cron format is the only option if I want this to run more frequently than once an hour, right?
You can modify a configuration file to add more listbox options but what is the point? You know cron syntax so just use that. It is all cron under the hood in