Deployment Architecture

How should I set the con_replication_max_pull_count value for search head cluster members to pull configuration changes from the captain?

ben_leung
Builder
WARN  ConfMetrics - single_action=PULL_FROM took wallclock_ms=4610! Consider a lower value of conf_replication_max_pull_count in server.conf on all members

What should I base the value on for conf_replication_max_pull_count? The warning is telling me that the cluster nodes are taking too long to pull configuration changes from the captain. Is my understanding correct?

conf_replication_max_pull_count = <int>
* Controls the maximum number of configuration changes a member will
  replicate from the captain at one time.
* A value of 0 disables any size limits.
* Defaults to 1000.

splunkIT
Splunk Employee
Splunk Employee

Unless advised by Support, it's probably not a good idea to modify the conf_replication_max_pull_count setting.

The WARN itself is not necessarily a problem, unless it corresponds to slow UI response times and/or general system problems.

In general, note that this message is based on wallclock time. That means any performance problem on the system – e.g. memory pressure or contention for CPU – can cause this WARN. It isn't always a problem with the configuration replication workload itself.

If the WARN message corresponds to slow UI response times and/or general system problems, then please contact Support and provide the following artifacts for further analysis:

1.) Collect new diags from captain and from at least one of the member nodes
2.) On each of the search heads, please take of backup of the latest bundle file under var/run/splunk/snapshot to a temporary directory, and provide them as well

0 Karma

splunkIT
Splunk Employee
Splunk Employee

What is your cluster's ref factor?

0 Karma

ben_leung
Builder

replication_factor = 1

but that is only for replication of search artifacts

The WARN message is referring to configuration changes, like knowledge objects changing by users via the UI.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...