Splunk Search
Highlighted

Real Time Search Issues

New Member

We're running into an issue where are RT searches are being delayed due to the amount of concurrent searches being ran.

priority=default, status=delayed, reason="The maximum number of concurrent running jobs for this real-time scheduled search on this instance has been reached", concurrencycategory="real-timescheduled", concurrencycontext="saved-searchinstance-wide", concurrencylimit=1, scheduledtime=1556040360, window_time=0

I have double checked everything in my limits.conf that could stop these searches.

basemaxsearches = 7
maxrtsearchmultiplier = 4
max
searchespercpu = 4 (4 cpu 18gb ram)

maxsearchesperc = 100
autosummaryperc = 100

We're on version 7.2.4.2 - cron alerts fire with no issues but RT do not. I know several people will say dont use RT alerts - not interested in your opinion in that regards - just whats holing up my own RT searches.

As you can see below - we dont have that many searches running.

alt text

0 Karma
Highlighted

Re: Real Time Search Issues

Esteemed Legend

I suggest that you give up on real-time searches for exactly this reason, among many other good ones: https://answers.splunk.com/answers/734767/why-are-realtime-searches-disliked-in-the-splunk-w.html

0 Karma
Highlighted

Re: Real Time Search Issues

Esteemed Legend

You cannot have more RT searches than CPU cores, but you can cheat and use fake RT:
On Search Heads in limits.conf:

[realtime]
#https://docs.splunk.com/Documentation/Splunk/latest/Search/Aboutrealtimesearches#Indexed_real-time_search
indexed_realtime_use_by_default = true
0 Karma