I have updated a Splunk Cluster from V7.3.8 to V8.1.2 following the documentation provided by Splunk and since the update we have an issue with the scheduled searches
Schedule searches are running normally after a Searchhead Cluster restart but after some time they skipping on the Capitan and and they do not run at all on the other nodes
In the screenshots above Scheduled Searches where running until 8 AM CET and then all are skipped on the Captain and the other Search Heads did not process any Scheduled Searches.
I found a workaround to move the Captain to another SearchHead and then Schedules Searches will run again. As seen in the example above
The cluster is composed of 3 Indexers, 3 SearchHeads and 1 Master node
I have increased the Relative concurrency limit for scheduled searches to 70% and Relative concurrency limit for scheduled searches to the same 70%
Also adapted the limits.conf to
# The base number of concurrent searches.
base_max_searches = 60
# Max real-time searches = max_rt_search_multiplier x max historical searches.
# max_rt_search_multiplier = 1
# The maximum number of concurrent searches per CPU.
max_searches_per_cpu = 10
max_searches_perc = 60
But nothing helps
A sure way to reproduce this on the system is to stop one of the SearchHeads and then start it. Aprox 10 Minutes after the SearchHead starts all scheduled searches will be skipped on the Capitan
In the Logs there is only one type of "Error" (actually info message) :
|_ACCELERATE_AF2AEFDE-8E13-4DCA-90CB-C21D356D9A60_iqpress_nobody_e0c3b6f1a41c2518_ACCELERATE_||The maximum number of concurrent historical scheduled searches on this cluster has been reached (220)|
Thank you very much in advance 🙂
Some questions to you: