Can you verify my plans for a search head cluster configuration?

[allan_newton]

Hi All,

I'm trying to create a sh cluster, here are the sequential things that I have. Please correct me.

On the deployer
[shclustering]
pass4SymmKey = shc@cluster
shcluster_label = sh_cluster

restart the deployer

On all the search heads except deployer
splunk init shcluster-config -auth admin:changeme -mgmt_uri https://respective_sh_ip:mgmt_port -replication_port rep_port -replication_factor 2 -conf_deploy_fetch_url https://deployer_ip:mgmt_port -secret shc@cluster -shcluster_label sh_cluster

restart the search heads after the configuration.

Only on one search head and not deployer
splunk bootstrap shcluster-captain -servers_list "https://sh1_ip:mgmt_port,https://sh2_ip:mgmt_port,https://sh3_ip:mgmt_port" -auth admin:admin

Push configurations from deployer to sh member.

create a app and moved it to etc/shcluster on deployer, then

splunk apply shcluster-bundle -target https://sh1_ip:mgmt_port

Thanks,
Allan

[koshyk]

better to put the above CLI command parameters (eg replication_factor etc.) into .conf files, so that you can version control and automate

[kunalmao · Accepted solution]

Everything looks fine, you will have to incorporate comment made by jdosch1 if you haven't , apart from that i would suggest you test your search head cluster by taking the captain down and then running ./splunk shcluster-status to see if election of new captain is being done smoothly before applying changes from deployer.

Also add the below stanza in server.conf of all the members to ensure that your search head captain is not loaded with scheduled searches and is performing optimally

[shclustering]
captain_is_adhoc_searchhead = true

[skoelpin]

Looks correct to me.

The one gotcha with SHC is that all local folders will be merged with default on the deployer

During the staging process that occurs prior to pushing the configuration bundle, the deployer copies the configuration bundle to a staging area on its file system, where it merges all settings from files in /shcluster/apps/<appname>/local into corresponding files in /shcluster/apps/<appname>/default. The deployer then pushes only the merged default files.

http://docs.splunk.com/Documentation/Splunk/7.0.0/DistSearch/PropagateSHCconfigurationchanges

[jdosch1]

Looks good.
have you already configured the SH to contact the Cluster Master ? If not you need to run the CLI below
./splunk edit cluster-config -mode searchhead -master_uri https://clustermaster.dev.com:8089 -secret

[richgalloway]

Looks OK to me.