Splunk Search

Real Time Search Issues

shanecifaldi
New Member

We're running into an issue where are RT searches are being delayed due to the amount of concurrent searches being ran.

priority=default, status=delayed, reason="The maximum number of concurrent running jobs for this real-time scheduled search on this instance has been reached", concurrency_category="real-time_scheduled", concurrency_context="saved-search_instance-wide", concurrency_limit=1, scheduled_time=1556040360, window_time=0

I have double checked everything in my limits.conf that could stop these searches.

base_max_searches = 7
max_rt_search_multiplier = 4
max_searches_per_cpu = 4 (4 cpu 18gb ram)

max_searches_perc = 100
auto_summary_perc = 100

We're on version 7.2.4.2 - cron alerts fire with no issues but RT do not. I know several people will say dont use RT alerts - not interested in your opinion in that regards - just whats holing up my own RT searches.

As you can see below - we dont have that many searches running.

alt text

0 Karma

woodcock
Esteemed Legend

You cannot have more RT searches than CPU cores, but you can cheat and use fake RT:
On Search Heads in limits.conf:

[realtime]
#https://docs.splunk.com/Documentation/Splunk/latest/Search/Aboutrealtimesearches#Indexed_real-time_search
indexed_realtime_use_by_default = true
0 Karma

woodcock
Esteemed Legend

I suggest that you give up on real-time searches for exactly this reason, among many other good ones: https://answers.splunk.com/answers/734767/why-are-realtime-searches-disliked-in-the-splunk-w.html

0 Karma