Splunk Enterprise

Working with Search Head cluster - Replication issue

vijaysri
Builder

Hi,

I have created a KVstore in Search Head deployer, that KVstore is not replicated to Search heads.

The below setting is given as "true" in Search Head deployer.

conf_replication_include.lookups = true

What else need to be changed?

 

Labels (1)
0 Karma
1 Solution

lakshman239
SplunkTrust
SplunkTrust

Not sure what you mean by 'Search Heads are not reported to deployer'.?

I assume you are are setting up the deployer and SHC from scratch as per the doc - https://docs.splunk.com/Documentation/Splunk/8.2.0/DistSearch/SHCdeploymentoverview 

The only thing I notice is you are using http instead of https in conf_deploy_fetch_url http://deployerIPaddress:8089. Is the deployer not running https?

is the SHC status healthy? whats the output of the kvstatus command?

If you create a lookup in one of the SHC member via UI, does that get replicated to the other 2 members? If so, replication of lookups/knowledge objects works [ you can test for dashboards etc..]

Have you then followed up the doc to connect to cluster master/indexers?

 

View solution in original post

0 Karma

vijaysri
Builder

Hi @lakshman239 

KVstore is working fine in the deployer.

Search Heads are not reported to deployer, I have followed the below steps even after that, search heads are not reporting.

 

In deployer --> server.conf

[shclustering]
pass4SymmKey = passkey
shcluster_label = shcluster1

In Search Heads - 3 search Heads

./splunk init shcluster-config -auth admin:password -mgmt_uri https://SH1-IPaddress:8089 -replication_port 34567 -replication_factor 3 -conf_deploy_fetch_url http://deployerIPaddress:8089 -secret passkey -shcluster_label shcluster1

./splunk restart


./splunk init shcluster-config -auth admin:password -mgmt_uri https://SH2-IPaddress:8089 -replication_port 34567 -replication_factor 3 -conf_deploy_fetch_url http://deployerIPaddress:8089 -secret passkey -shcluster_label shcluster1

./splunk restart

./splunk init shcluster-config -auth admin:password -mgmt_uri https://SH3-IPaddress:8089 -replication_port 34567 -replication_factor 3 -conf_deploy_fetch_url http://deployerIPaddress:8089 -secret passkey -shcluster_label shcluster1
./splunk restart


./splunk bootstrap shcluster-captain -servers_list "https://SH1-IPaddress:8089,https://SH2-IPaddress:8089,https://SH3-IPaddress:8089" -auth admin:password

./splunk show shcluster-status -auth admin:password

./splunk show kvstore-status -auth admin:password

0 Karma

lakshman239
SplunkTrust
SplunkTrust

Not sure what you mean by 'Search Heads are not reported to deployer'.?

I assume you are are setting up the deployer and SHC from scratch as per the doc - https://docs.splunk.com/Documentation/Splunk/8.2.0/DistSearch/SHCdeploymentoverview 

The only thing I notice is you are using http instead of https in conf_deploy_fetch_url http://deployerIPaddress:8089. Is the deployer not running https?

is the SHC status healthy? whats the output of the kvstatus command?

If you create a lookup in one of the SHC member via UI, does that get replicated to the other 2 members? If so, replication of lookups/knowledge objects works [ you can test for dashboards etc..]

Have you then followed up the doc to connect to cluster master/indexers?

 

0 Karma

lakshman239
SplunkTrust
SplunkTrust

Hi @vijaysri,  we don't need to explicitly define conf_replication_include.lookups = true, as this is already defined in etc/system/default/server.conf .

You would need to ensure the collections.conf and transforms.conf have the correct/required conf - Have a look at the docs and https://dev.splunk.com/enterprise/docs/developapps/manageknowledge/kvstore/usingconfigurationfiles/ 

0 Karma
Get Updates on the Splunk Community!

Splunk Education - Fast Start Program!

Welcome to Splunk Education! Splunk training programs are designed to enable you to get started quickly and ...

Five Subtly Different Ways of Adding Manual Instrumentation in Java

You can find the code of this example on GitHub here. Please feel free to star the repository to keep in ...

New Splunk APM Enhancements Help Troubleshoot Your MySQL and NoSQL Databases Faster

Splunk Observability has two new enhancements to make it quicker and easier to troubleshoot slow or frequently ...