Splunk Search
Highlighted

Why are real time searches not running and getting error "status skipped, reason="Real time searches pending""?

Communicator

Real time searches are not running, and searching for one of the saved search names in the _internal index shows:

status=skipped, reason="Real time searches pending"

What does "Real time searches pending mean"? What are possible causes of this?

This is on Splunk Enterprise 6.4.1 and I do not see anything in the _internal index relating to the concurrent search limit being met.

0 Karma
Highlighted

Re: Why are real time searches not running and getting error "status skipped, reason="Real time searches pending""?

Motivator

Who's running those realtime searches? Does the user/role has the ability to run realtime searches? If not, you will see an error stating "Skipping searches as xyz does not have permissions etc". Take a look at the scheduler.log and find out.

Other reason would be, if there's not enough horse power on the search heads.

Hope this helps!

Thanks,
Raghav

0 Karma
Highlighted

Re: Why are real time searches not running and getting error "status skipped, reason="Real time searches pending""?

Contributor

it is in the Palo Alto app, this was working in 6.4.1.

0 Karma
Highlighted

Re: Why are real time searches not running and getting error "status skipped, reason="Real time searches pending""?

Esteemed Legend

Each realtime search unpreemptively locks 1 core on EVERY INDEXER and on your Search Head. If you have more realtime searches than cores, you will get this error. If you are running this many realtime searches, you are looking for trouble and can almost surely be doing something nicer/better.

Highlighted

Re: Why are real time searches not running and getting error "status skipped, reason="Real time searches pending""?

Path Finder

Here's my question: What is the best way to convert a real-time alert to something more manageable, and yet maintain a similar net effect?

0 Karma
Highlighted

Re: Why are real time searches not running and getting error "status skipped, reason="Real time searches pending""?

Esteemed Legend

Schedule it to cover a span of X and run it every X/2. This covers the case where events at the end of span t an the beginning of t+1 would just miss triggering in those windows but will hit in the next alert run. Then make X as large as you can stomach.

0 Karma
Highlighted

Re: Why are real time searches not running and getting error "status skipped, reason="Real time searches pending""?

Path Finder

Do you suggest doing the cron scheduling format? One of these is a situation where we're monitoring account lockouts, so our need is for it to be accurate enough to act upon -- say 5 minutes. What about duplicates?

0 Karma
Highlighted

Re: Why are real time searches not running and getting error "status skipped, reason="Real time searches pending""?

Esteemed Legend

The only difference in the cron format is that it gives you complete granularity.
Cron for every 5 minutes is */5 * * * *. Alert once per user and throttle accordingly with the throttling settings.

0 Karma
Highlighted

Re: Why are real time searches not running and getting error "status skipped, reason="Real time searches pending""?

Path Finder

Unless I'm missing something, cron format is the only option if I want this to run more frequently than once an hour, right?

0 Karma
Highlighted

Re: Why are real time searches not running and getting error "status skipped, reason="Real time searches pending""?

Esteemed Legend

You can modify a configuration file to add more listbox options but what is the point? You know cron syntax so just use that. It is all cron under the hood in savedsearches.conf anyway.

0 Karma