Deployment Architecture

How to migrate KV store data from a search head standalone to a search head cluster ?

New Member


I have a standalone search head with KVstores.
I want to migrate the KVstores to a search head cluster without, if possible, exporting all data (in csv or other format) and importing them again as it represents a large quantity of data (2-3GB) and many collections.

What I tryed :

  • backup the kvstores from the standalone server using
    ./splunk backup kvstore

  • Set the replication factor to 1 on one search head of the new cluster

  • Clean kvstore db on this search head :
    ./splunk clean kvstore --local
    ./splunk clean kvstore --cluster

  • Restore on the clustered SH the backuped kvstore from archive
    ./splunk restore kvstore archiveName
    This step took a very long time (maybe its normal).

  • I monitored this using
    ./splunk show shcluster-status

  • The backupRestoreStatus finally moved to ready :

This member:
backupRestoreStatus : Ready
date : Fri Nov 29 13:34:12 2019
dateSec : 1575034452.206
disabled : 0
guid : 0C76D3C2-F11A-47FB-A705-3ECBC0CCE929
oplogEndTimestamp : Fri Nov 29 13:34:05 2019
oplogEndTimestampSec : 1575034445
oplogStartTimestamp : Fri Nov 29 10:11:49 2019
oplogStartTimestampSec : 1575022309
port : 8191
replicaSet : splunkrs
replicationStatus : KV store captain
standalone : 0
status : ready

Enabled KV store members:
guid : 0C76D3C2-F11A-47FB-A705-3ECBC0CCE929
hostAndPort : sh01:8191

KV store members:
configVersion : 1
electionDate : Fri Nov 29 13:24:26 2019
electionDateSec : 1575033866
hostAndPort : spplsh01:8191
optimeDate : Fri Nov 29 13:34:05 2019
optimeDateSec : 1575034445
replicationStatus : KV store captain
uptime : 608

But even if the kvstore status is all ok, when I search for data in the kvstores these are empty (even if there are lot of files in the mongo directory).
As this step is not ok, of course, I cannot go further trying to sync with another search head.

Has anyone already tried to do this ? maybe using another method ? for next steps, do I need to do the same on all SH of cluster or will the kvstores replicate automaticaly ?

Thanks in advance.

The used Splunk version is 7.3.2

0 Karma
Get Updates on the Splunk Community!

New Learning Videos on Topics Most Requested by You! Plus This Month’s New Splunk ...

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

How I Instrumented a Rust Application Without Knowing Rust

As a technical writer, I often have to edit or create code snippets for Splunk's distributions of ...

Splunk Community Platform Survey

Hey Splunk Community, Starting today, the community platform may prompt you to participate in a survey. The ...