Real Time Search Issues

[shanecifaldi]

We're running into an issue where are RT searches are being delayed due to the amount of concurrent searches being ran.

priority=default, status=delayed, reason="The maximum number of concurrent running jobs for this real-time scheduled search on this instance has been reached", concurrency_category="real-time_scheduled", concurrency_context="saved-search_instance-wide", concurrency_limit=1, scheduled_time=1556040360, window_time=0

I have double checked everything in my limits.conf that could stop these searches.

base_max_searches = 7
max_rt_search_multiplier = 4
max_searches_per_cpu = 4 (4 cpu 18gb ram)

max_searches_perc = 100
auto_summary_perc = 100

We're on version 7.2.4.2 - cron alerts fire with no issues but RT do not. I know several people will say dont use RT alerts - not interested in your opinion in that regards - just whats holing up my own RT searches.

As you can see below - we dont have that many searches running.

[woodcock]

You cannot have more RT searches than CPU cores, but you can cheat and use fake RT:
On Search Heads in limits.conf:

[realtime]
#https://docs.splunk.com/Documentation/Splunk/latest/Search/Aboutrealtimesearches#Indexed_real-time_search
indexed_realtime_use_by_default = true

[woodcock]

I suggest that you give up on real-time searches for exactly this reason, among many other good ones: https://answers.splunk.com/answers/734767/why-are-realtime-searches-disliked-in-the-splunk-w.html