We're running into an issue where are RT searches are being delayed due to the amount of concurrent searches being ran.
priority=default, status=delayed, reason="The maximum number of concurrent running jobs for this real-time scheduled search on this instance has been reached", concurrencycategory="real-timescheduled", concurrencycontext="saved-searchinstance-wide", concurrencylimit=1, scheduledtime=1556040360, window_time=0
I have double checked everything in my limits.conf that could stop these searches.
basemaxsearches = 7
maxrtsearchmultiplier = 4
maxsearchespercpu = 4 (4 cpu 18gb ram)
maxsearchesperc = 100
autosummaryperc = 100
We're on version 126.96.36.199 - cron alerts fire with no issues but RT do not. I know several people will say dont use RT alerts - not interested in your opinion in that regards - just whats holing up my own RT searches.
As you can see below - we dont have that many searches running.
You cannot have more RT searches than CPU cores, but you can cheat and use fake RT:
On Search Heads in
[realtime] #https://docs.splunk.com/Documentation/Splunk/latest/Search/Aboutrealtimesearches#Indexed_real-time_search indexed_realtime_use_by_default = true