Deployment Architecture
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Ask a Question
Solved! Jump to solution

After upgrading to 6.5.0, why is search head cluster skipping about 50% of scheduled searches?

rbal_splunk
Splunk Employee
Splunk Employee
‎10-31-2016 11:13 AM

On 10 Node SHC deployment – post upgrade from 6.2.5 to 6.5.0 system, instance is skipping about 50% of the scheduled searches.

Reply
1 Solution
Solved! Jump to solution
Solution

rbal_splunk
Splunk Employee
Splunk Employee
‎10-31-2016 11:16 AM

This issue has been resolved and the following steps were taken to debug and resolve the issue:

1)Observation 1: The following search showed that SHC members the delegatejob was taking up to 200seconds

Index=_internal source=*splunkd_access.log delegatejob | stats avg(spent) by host

2)Observation 2: It was seen that on SHC members ps -ef | grep splunk | search, a lot of Splunk launcher jobs were hanging.

To resolve the issue, the following changes were implemented:

On All SHC members implemented:

$SPLUNK_HOME/etc/system/local/limits.conf 

[search] 
search_process_mode = traditional

After above changes were made the skipping searches frequency has dropped significantly

View solution in original post

Reply
Solved! Jump to solution
Solution

rbal_splunk
Splunk Employee
Splunk Employee
‎10-31-2016 11:16 AM

This issue has been resolved and the following steps were taken to debug and resolve the issue:

1)Observation 1: The following search showed that SHC members the delegatejob was taking up to 200seconds

Index=_internal source=*splunkd_access.log delegatejob | stats avg(spent) by host

2)Observation 2: It was seen that on SHC members ps -ef | grep splunk | search, a lot of Splunk launcher jobs were hanging.

To resolve the issue, the following changes were implemented:

On All SHC members implemented:

$SPLUNK_HOME/etc/system/local/limits.conf 

[search] 
search_process_mode = traditional

After above changes were made the skipping searches frequency has dropped significantly

Reply
Solved! Jump to solution

onthebay
Path Finder
‎06-08-2017 10:17 AM

What is the reasoning behind why this resolves the issue? What has changed with the auto setting in 6.5?

0 Karma
Reply
Get Updates on the Splunk Community!

Now Available: Cisco Talos Threat Intelligence Integrations for Splunk Security Cloud ...

At .conf24, we shared that we were in the process of integrating Cisco Talos threat intelligence into Splunk ...

Preparing your Splunk Environment for OpenSSL3

The Splunk platform will transition to OpenSSL version 3 in a future release. Actions are required to prepare ...

Easily Improve Agent Saturation with the Splunk Add-on for OpenTelemetry Collector

Agent Saturation What and Whys In application performance monitoring, saturation is defined as the total load ...