When adding a new search head member to an existing cluster, I cannot for the life of me get the Kvstore to initialize

[briancronrath]

In the splunkd.log I can see the following: 

"Local KV Store has replication issues. See introspection data and mongod.log for details. Cluster has not been configured on this member. KVStore cluster has not been configured."

If I check the kvstore-status it just says the kvstore status is down for this new member.  The normal shcluster-status shows as UP and the kvstore as ready however for this new member.  Not sure what I can do to try and force the kvstore to initialize?  I have tried shutting splunk down on the new member and doing a kvstore-clean and restarting but it still isn't taking.

Splunk version 9.0.0

Any thoughts on what else I can try?

[PickleRick]

Since one of main points of the search-head cluster is to have stuff replicated, in case of problems with a single node it's often easiest to scratch it completely, reinstall and add to the cluster as a fresh node.

[isoutamo]

Hi

as @PickleRick said, remove whole splunk installation (e.g. yum remove splunk) from this node and use also rm -fr /opt/splunk. Then just install it again and follow up this https://docs.splunk.com/Documentation/Splunk/9.0.4/DistSearch/Addaclustermember

That is the way how I do it if there are any issues with new node.

And remember you cannot/shouldn't try to add any used instance into SHC!

r. Ismo

[PickleRick]

Yep +1 on that. I might not have been clear enough on that point. The instance added to a SH cluster should be completely "clean".