Why are there errors when configuring search head clustering with a deployer?

[Prakash493]

Hi , I am configuring a search head clustering with a deployer but ran in many issues:
as per the Splunk docs these are the steps I did :

First I set the search head deployer:
I have added a pass4symmkey on deployer under shcluster stanza

[general]
serverName = sh2
pass4SymmKey = $1$ir3GnxLQSyQCIHmqN+mx
[shclustering]
pass4SymmKey = $1$1WrJE7i8jQ+THZ0MWxYh

I kept this same key across all search heads on server.conf ([shclustering]

pass4SymmKey = $1$1WrJE7i8jQ+THZ0MWxYh) 

Setting up search head cluster:
I ran this command in all search heads (total 3):

./splunk init shcluster-config -auth admin:password -mgmt_uri "https://vvvvv:8089" -replication_port 8079 -replication_factor 3 -conf_deploy_fetch_url https:/// (deployer_ip_address):8089 -secret  -shcluster_label 

I run this command in all search heads (Total 3)
then I did restart.

Choosing captain:
after completing all the above steps I choose any one search head to make a captain;

Then I run this command :

./splunk bootstrap shcluster-captain -servers_list "URI:8089". -autgh admin:password

But after running its saying ,

error=This node seems to have already joined another cluster with below members: 'https://xxxx:8089'. First remove the member from the old cluster. Then run 'splunk clean
 raft' on the member to reuse it in a new cluster; server=https://xxxxx:8089, error=This node seems to have already joined another cluster with below members: 'https://xxxxx:8089'. First remove the me
mber from the old cluster. Then run 'splunk clean raft' on the member to reuse it in a new cluster.

I am not clear why I am getting this error first time I am setting on these new servers , when I am running to make a captain getting this error. Please help me

2) And what the difference between general pass4symmkey and shcluster pass4symmkey , my pass4symmkey the shclustering one is same across all cluster members not the pass4symmkey which is under general stanza , which one should need to same across all search heads ?

Thanks

[VatsalJagani]

@Prakash493,
Few points that you need to notice.

• You should not update passSymmKey under [general]
• You should not add encrypted passSymmKey under [shcluster], instead you need to add passKey in normal characters and do splunk restart, splunk will encrypt that.
• Your searchhead cluster need also to be connected with indexer cluster for that you need to add [clustering] same as you might have done on indexer.
• Rest steps are okay about choosing captain and all.

Check if you did something wrong and redeploy the shcluster.

[lakshman239]

Not sure if your issue is resolved. I assume you are following steps as per https://docs.splunk.com/Documentation/Splunk/7.2.6/DistSearch/SHCdeploymentoverview

• pls note that the security_key (pass4SymmKey) under shclustering stanza is used and should be same both in your deployer and each member of the SHC. The key in general is auto-generated as part of splunk install.
• I suggest, clean and re-initialize all the instances and follow the steps again and use splunk show shcluster-status to check the status of the cluster after boot-strap.