Splunk Search

Real Time Search Issues

New Member

We're running into an issue where are RT searches are being delayed due to the amount of concurrent searches being ran.

priority=default, status=delayed, reason="The maximum number of concurrent running jobs for this real-time scheduled search on this instance has been reached", concurrency_category="real-time_scheduled", concurrency_context="saved-search_instance-wide", concurrency_limit=1, scheduled_time=1556040360, window_time=0

I have double checked everything in my limits.conf that could stop these searches.

base_max_searches = 7
max_rt_search_multiplier = 4
max_searches_per_cpu = 4 (4 cpu 18gb ram)

max_searches_perc = 100
auto_summary_perc = 100

We're on version - cron alerts fire with no issues but RT do not. I know several people will say dont use RT alerts - not interested in your opinion in that regards - just whats holing up my own RT searches.

As you can see below - we dont have that many searches running.

alt text

0 Karma

Esteemed Legend

You cannot have more RT searches than CPU cores, but you can cheat and use fake RT:
On Search Heads in limits.conf:

indexed_realtime_use_by_default = true
0 Karma

Esteemed Legend

I suggest that you give up on real-time searches for exactly this reason, among many other good ones: https://answers.splunk.com/answers/734767/why-are-realtime-searches-disliked-in-the-splunk-w.html

0 Karma