topic Working with Search Head cluster - Replication issue in Splunk Enterprise

Working with Search Head cluster - Replication issue

VijaySrrie — Tue, 18 May 2021 04:21:53 GMT

Hi,

I have created a KVstore in Search Head deployer, that KVstore is not replicated to Search heads.

The below setting is given as "true" in Search Head deployer.

conf_replication_include.lookups = true

What else need to be changed?

Re: Working with Search Head cluster - Replication issue

lakshman239 — Tue, 18 May 2021 08:52:24 GMT

Hi @VijaySrrie, we don't need to explicitly define conf_replication_include.lookups = true, as this is already defined in etc/system/default/server.conf .

You would need to ensure the collections.conf and transforms.conf have the correct/required conf - Have a look at the docs and https://dev.splunk.com/enterprise/docs/developapps/manageknowledge/kvstore/usingconfigurationfiles/

Re: Working with Search Head cluster - Replication issue

VijaySrrie — Wed, 19 May 2021 00:37:58 GMT

Hi @lakshman239

KVstore is working fine in the deployer.

Search Heads are not reported to deployer, I have followed the below steps even after that, search heads are not reporting.

In deployer --> server.conf

[shclustering]
pass4SymmKey = passkey
shcluster_label = shcluster1

In Search Heads - 3 search Heads

./splunk init shcluster-config -auth admin:password -mgmt_uri https://SH1-IPaddress:8089 -replication_port 34567 -replication_factor 3 -conf_deploy_fetch_url http://deployerIPaddress:8089 -secret passkey -shcluster_label shcluster1

./splunk restart

./splunk init shcluster-config -auth admin:password -mgmt_uri https://SH2-IPaddress:8089 -replication_port 34567 -replication_factor 3 -conf_deploy_fetch_url http://deployerIPaddress:8089 -secret passkey -shcluster_label shcluster1

./splunk restart

./splunk init shcluster-config -auth admin:password -mgmt_uri https://SH3-IPaddress:8089 -replication_port 34567 -replication_factor 3 -conf_deploy_fetch_url http://deployerIPaddress:8089 -secret passkey -shcluster_label shcluster1
./splunk restart

./splunk bootstrap shcluster-captain -servers_list "https://SH1-IPaddress:8089,https://SH2-IPaddress:8089,https://SH3-IPaddress:8089" -auth admin:password

./splunk show shcluster-status -auth admin:password

./splunk show kvstore-status -auth admin:password

Re: Working with Search Head cluster - Replication issue

lakshman239 — Wed, 19 May 2021 09:29:22 GMT

Not sure what you mean by 'Search Heads are not reported to deployer'.?

I assume you are are setting up the deployer and SHC from scratch as per the doc - https://docs.splunk.com/Documentation/Splunk/8.2.0/DistSearch/SHCdeploymentoverview

The only thing I notice is you are using http instead of https in conf_deploy_fetch_url http://deployerIPaddress:8089. Is the deployer not running https?

is the SHC status healthy? whats the output of the kvstatus command?

If you create a lookup in one of the SHC member via UI, does that get replicated to the other 2 members? If so, replication of lookups/knowledge objects works [ you can test for dashboards etc..]

Have you then followed up the doc to connect to cluster master/indexers?