In the splunkd.log I can see the following:
"Local KV Store has replication issues. See introspection data and mongod.log for details. Cluster has not been configured on this member. KVStore cluster has not been configured."
If I check the kvstore-status it just says the kvstore status is down for this new member. The normal shcluster-status shows as UP and the kvstore as ready however for this new member. Not sure what I can do to try and force the kvstore to initialize? I have tried shutting splunk down on the new member and doing a kvstore-clean and restarting but it still isn't taking.
Splunk version 9.0.0
Any thoughts on what else I can try?
Since one of main points of the search-head cluster is to have stuff replicated, in case of problems with a single node it's often easiest to scratch it completely, reinstall and add to the cluster as a fresh node.
Hi
as @PickleRick said, remove whole splunk installation (e.g. yum remove splunk) from this node and use also rm -fr /opt/splunk. Then just install it again and follow up this https://docs.splunk.com/Documentation/Splunk/9.0.4/DistSearch/Addaclustermember
That is the way how I do it if there are any issues with new node.
And remember you cannot/shouldn't try to add any used instance into SHC!
r. Ismo
Yep +1 on that. I might not have been clear enough on that point. The instance added to a SH cluster should be completely "clean".