Why are real time searches not running and getting error "status skipped, reason="Real time searches pending""?

[gn694]

Real time searches are not running, and searching for one of the saved search names in the _internal index shows:

status=skipped, reason="Real time searches pending"

What does "Real time searches pending mean"? What are possible causes of this?

This is on Splunk Enterprise 6.4.1 and I do not see anything in the _internal index relating to the concurrent search limit being met.

[Jarohnimo]

When is ok to disable one of these offending real time searches? I know for ITSI it needs to be installed on both Search and indexer but i'm not sure why on the indexer ITSI event grouping (noteable events) always gets skipped but not on the WFE.

I wish i could just turn it off on the Indexer to bypass all these skipped events but i'm unsure if doing this will break something on the search head.

[Jarohnimo]

I disabled the real time search around noon today and noticed the skipped searches ratio was already down to 20%

I know Enterprise Security is their flagship program but please support ITSI a little better. I'm starting to feel I know more than most of their premier Splunk engineers. I had to once wait a few weeks for a senior engineer to go to ITSI class to learn the tool prior to helping me.

The support for this program is very low and I hope they don't get rid of it

[woodcock]

The ITSI stuff is a different thing. That log is technically a bug and is probably fixed i the latest releases. While it is true that the real-time (ITSI or otherwise) alerts are "skipped" by the scheduler, the only reason that this is so is because they are never supposed to stop and so never actually need to be restarted. So for real-time, these logs should be ignored completely (and will eventually go away when you upgrade). As far as disabling, I would just try it on a Friday and check everything on Saturday and if it is all good, leave it off.

[woodcock]

Each realtime search unpreemptively locks 1 core on EVERY INDEXER and on your Search Head. If you have more realtime searches than cores, you will get this error. If you are running this many realtime searches, you are looking for trouble and can almost surely be doing something nicer/better.

[jhillenburg]

Here's my question: What is the best way to convert a real-time alert to something more manageable, and yet maintain a similar net effect?

[woodcock]

Schedule it to cover a span of X and run it every X/2. This covers the case where events at the end of span t an the beginning of t+1 would just miss triggering in those windows but will hit in the next alert run. Then make X as large as you can stomach.

[jhillenburg]

Do you suggest doing the cron scheduling format? One of these is a situation where we're monitoring account lockouts, so our need is for it to be accurate enough to act upon -- say 5 minutes. What about duplicates?

[woodcock]

The only difference in the cron format is that it gives you complete granularity.
Cron for every 5 minutes is */5 * * * *. Alert once per user and throttle accordingly with the throttling settings.

[jhillenburg]

Unless I'm missing something, cron format is the only option if I want this to run more frequently than once an hour, right?

[woodcock]

You can modify a configuration file to add more listbox options but what is the point? You know cron syntax so just use that. It is all cron under the hood in savedsearches.conf anyway.

[jhillenburg]

Thanks. That has helped a lot.

[woodcock]

If we are all good here, be sure to click Accept to close the question (and UpVote any helpful comments).

[Raghav2384]

Who's running those realtime searches? Does the user/role has the ability to run realtime searches? If not, you will see an error stating "Skipping searches as xyz does not have permissions etc". Take a look at the scheduler.log and find out.

Other reason would be, if there's not enough horse power on the search heads.

Hope this helps!

Thanks,
Raghav

[sbattista09]

it is in the Palo Alto app, this was working in 6.4.1.