Splunk Enterprise Searchead Cluster not running scheduled searches

[Octavian]

Hello,

I have updated a Splunk Cluster from V7.3.8 to V8.1.2 following the documentation provided by Splunk and since the update we have an issue with the scheduled searches

Schedule searches are running normally after a Searchhead Cluster restart but after some time they skipping  on the Capitan and and they do not run at all on the other nodes

In the screenshots above Scheduled Searches where running until  8 AM CET and then all are skipped on the Captain and the other Search Heads did not process any Scheduled Searches.

I found a workaround to move the Captain to another SearchHead and then Schedules Searches will run again.  As seen in the example above

The cluster is composed of 3 Indexers, 3 SearchHeads and 1 Master node

I have increased the Relative concurrency limit for scheduled searches to 70% and Relative concurrency limit for scheduled searches to the same 70%

Also adapted the limits.conf to 

# The base number of concurrent searches.
base_max_searches = 60

# Max real-time searches = max_rt_search_multiplier x max historical searches.
# max_rt_search_multiplier = 1

# The maximum number of concurrent searches per CPU.
max_searches_per_cpu = 10

max_searches_perc = 60

But nothing helps

A sure way to reproduce this on the system is to stop one of the SearchHeads and then start it. Aprox 10 Minutes after the SearchHead starts all scheduled searches will be skipped on the Capitan

In the Logs there is only one type of "Error" (actually info message) : 

_ACCELERATE_AF2AEFDE-8E13-4DCA-90CB-C21D356D9A60_iqpress_nobody_e0c3b6f1a41c2518_ACCELERATE_

The maximum number of concurrent historical scheduled searches on this cluster has been reached (220)

Thank you very much in advance 🙂

[isoutamo]

Some questions to you:

• Have all SH nodes equal HW and other resources?
• Are there any stuck searches?
• Have you RT searches running one that SHC?
• Does this start after update and works before it?
• Are all nodes (LM, SHC, MC, CM, IDX-peers)  in your environment running the same splunk version?

r. Ismo

[Octavian]

Hello

• yes - all SH and IDX are running the same hardware (VM's with 16 Cores 55 GB Ram)
• no stuck searches 
• yes - there are realtime searches running
• yes - no such issues where noticed before
• yes - all nodes (SH 3 nodes, IDX 3 nodes and 1 master) are running the same Splunk 8.1.2 Version

Thank you