Installation

When adding a new search head member to an existing cluster, I cannot for the life of me get the Kvstore to initialize

briancronrath
Contributor

In the splunkd.log I can see the following: 

"Local KV Store has replication issues. See introspection data and mongod.log for details. Cluster has not been configured on this member. KVStore cluster has not been configured."

If I check the kvstore-status it just says the kvstore status is down for this new member.  The normal shcluster-status shows as UP and the kvstore as ready however for this new member.  Not sure what I can do to try and force the kvstore to initialize?  I have tried shutting splunk down on the new member and doing a kvstore-clean and restarting but it still isn't taking.

Splunk version 9.0.0

 

Any thoughts on what else I can try?

Labels (1)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

Since one of main points of the search-head cluster is to have stuff replicated, in case of problems with a single node it's often easiest to scratch it completely, reinstall and add to the cluster as a fresh node.

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

as @PickleRick said, remove whole splunk installation (e.g. yum remove splunk) from this node and use also rm -fr /opt/splunk. Then just install it again and follow up this https://docs.splunk.com/Documentation/Splunk/9.0.4/DistSearch/Addaclustermember

That is the way how I do it if there are any issues with new node.

And remember you cannot/shouldn't try to add any used instance into SHC!

r. Ismo

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Yep +1 on that. I might not have been clear enough on that point. The instance added to a SH cluster should be completely "clean".

0 Karma
Get Updates on the Splunk Community!

Observe and Secure All Apps with Splunk

  Join Us for Our Next Tech Talk: Observe and Secure All Apps with SplunkAs organizations continue to innovate ...

Splunk Decoded: Business Transactions vs Business IQ

It’s the morning of Black Friday, and your e-commerce site is handling 10x normal traffic. Orders are flowing, ...

Fastest way to demo Observability

I’ve been having a lot of fun learning about Kubernetes and Observability. I set myself an interesting ...