Deployment Architecture

After upgrading to 6.5.0, why is search head cluster skipping about 50% of scheduled searches?

rbal_splunk
Splunk Employee
Splunk Employee

On 10 Node SHC deployment – post upgrade from 6.2.5 to 6.5.0 system, instance is skipping about 50% of the scheduled searches.

1 Solution

rbal_splunk
Splunk Employee
Splunk Employee

This issue has been resolved and the following steps were taken to debug and resolve the issue:

1)Observation 1: The following search showed that SHC members the delegatejob was taking up to 200seconds

Index=_internal source=*splunkd_access.log delegatejob | stats avg(spent) by host

2)Observation 2: It was seen that on SHC members ps -ef | grep splunk | search, a lot of Splunk launcher jobs were hanging.

To resolve the issue, the following changes were implemented:

On All SHC members implemented:

$SPLUNK_HOME/etc/system/local/limits.conf

[search] 
search_process_mode = traditional 

After above changes were made the skipping searches frequency has dropped significantly

View solution in original post

rbal_splunk
Splunk Employee
Splunk Employee

This issue has been resolved and the following steps were taken to debug and resolve the issue:

1)Observation 1: The following search showed that SHC members the delegatejob was taking up to 200seconds

Index=_internal source=*splunkd_access.log delegatejob | stats avg(spent) by host

2)Observation 2: It was seen that on SHC members ps -ef | grep splunk | search, a lot of Splunk launcher jobs were hanging.

To resolve the issue, the following changes were implemented:

On All SHC members implemented:

$SPLUNK_HOME/etc/system/local/limits.conf

[search] 
search_process_mode = traditional 

After above changes were made the skipping searches frequency has dropped significantly

onthebay
Path Finder

What is the reasoning behind why this resolves the issue? What has changed with the auto setting in 6.5?

0 Karma
Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...