Knowledge Management

How to get existing KV Store to initialize after replacing one of the three (3) members with a new instance?

Path Finder

Splunkers,

Having trouble getting the kvstore to indicate that it is ready on any of the three members of the shcluster running Splunk 6.4.0 on CentOS 6.7.

There are 5 existing KV Stores and none of them can be accessed.

The trouble began when an overzealous admin accidentally deleted directories in one of one of our running shcluster members while it was running.

Attempted to use CLI commands to remove the corrupted member from one of the other members which seemed to work.

Then killed the Splunk related zombie processes left due to the pid file and bin directory being deleted the corrupted instance CLI could not be used.

Deleted the corrupted /opt/splunk instance then un-tar-ed another instance of Splunk 6.4.0 into a new /opt/splunk to replace the corrupted instance.

Followed the Splunk docks for "init" and "add new" to the shcluster. Once started issued CLI commands to make sure the new instance was properly configured.

The shcluster status is good and searches are possible from any shcluster member.

Attempt to do a simple search such as: | inputlookup

Yields errors indicating that the KV Store was not properly initialized.

If we had backups of the activity store data from the original three member shcluster a clean restart would make sense i.e. rebuild all the stores from scratch. We have files in folders contained in two of the three members, and can not access them via Splunk to create backup CSV files. We are hoping someone can guide us through getting the activity store initialized.

The mongod.log on the two remaining original shcluster members contain events such as:

Error in heartbeat request to ------------ InvalidReplicaSetConfig Our replica set configuration is invalid or does not include us

We have tried most of the non-destructive suggestions provided in Answers and Google searches.

THX

0 Karma
1 Solution

Path Finder

I was able to use the following to get the KV Store up and running:

https://answers.splunk.com/answers/375964/why-is-our-search-head-cluster-captain-generating.html

The problem is that the link does not provide an explanation for how to recover the backed up kvstore folders.

This is now what I need to find an answer to.

View solution in original post

Path Finder

I was able to use the following to get the KV Store up and running:

https://answers.splunk.com/answers/375964/why-is-our-search-head-cluster-captain-generating.html

The problem is that the link does not provide an explanation for how to recover the backed up kvstore folders.

This is now what I need to find an answer to.

View solution in original post

Path Finder

Hi Mdwecht,

Were you able to discover how to restore backed up data?

0 Karma