Reporting

Splunk Enterprise Searchead Cluster not running scheduled searches

Octavian
Observer

Hello,

I have updated a Splunk Cluster from V7.3.8 to V8.1.2 following the documentation provided by Splunk and since the update we have an issue with the scheduled searches

Schedule searches are running normally after a Searchhead Cluster restart but after some time they skipping  on the Capitan and and they do not run at all on the other nodes

Splunk 1.pngSplunk 2.png

In the screenshots above Scheduled Searches where running until  8 AM CET and then all are skipped on the Captain and the other Search Heads did not process any Scheduled Searches.

I found a workaround to move the Captain to another SearchHead and then Schedules Searches will run again.  As seen in the example above

The cluster is composed of 3 Indexers, 3 SearchHeads and 1 Master node

I have increased the Relative concurrency limit for scheduled searches to 70% and Relative concurrency limit for scheduled searches to the same 70%

Also adapted the limits.conf to 

# The base number of concurrent searches.
base_max_searches = 60

# Max real-time searches = max_rt_search_multiplier x max historical searches.
# max_rt_search_multiplier = 1

# The maximum number of concurrent searches per CPU.
max_searches_per_cpu = 10

max_searches_perc = 60

But nothing helps

 

A sure way to reproduce this on the system is to stop one of the SearchHeads and then start it. Aprox 10 Minutes after the SearchHead starts all scheduled searches will be skipped on the Capitan

 

In the Logs there is only one type of "Error" (actually info message) : 

_ACCELERATE_AF2AEFDE-8E13-4DCA-90CB-C21D356D9A60_iqpress_nobody_e0c3b6f1a41c2518_ACCELERATE_The maximum number of concurrent historical scheduled searches on this cluster has been reached (220)

 

Thank you very much in advance 🙂

 

 

Labels (1)
0 Karma

isoutamo
SplunkTrust
SplunkTrust

Some questions to you:

  • Have all SH nodes equal HW and other resources?
  • Are there any stuck searches?
  • Have you RT searches running one that SHC?
  • Does this start after update and works before it?
  • Are all nodes (LM, SHC, MC, CM, IDX-peers)  in your environment running the same splunk version?

r. Ismo

0 Karma

Octavian
Observer

Hello

 

  • yes - all SH and IDX are running the same hardware (VM's with 16 Cores 55 GB Ram)
  • no stuck searches 
  • yes - there are realtime searches running
  • yes - no such issues where noticed before
  • yes - all nodes (SH 3 nodes, IDX 3 nodes and 1 master) are running the same Splunk 8.1.2 Version

Thank you

0 Karma
Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...