Deployment Architecture

How to migrate KV store data from a search head standalone to a search head cluster ?

New Member


I have a standalone search head with KVstores.
I want to migrate the KVstores to a search head cluster without, if possible, exporting all data (in csv or other format) and importing them again as it represents a large quantity of data (2-3GB) and many collections.

What I tryed :

  • backup the kvstores from the standalone server using
    ./splunk backup kvstore

  • Set the replication factor to 1 on one search head of the new cluster

  • Clean kvstore db on this search head :
    ./splunk clean kvstore --local
    ./splunk clean kvstore --cluster

  • Restore on the clustered SH the backuped kvstore from archive
    ./splunk restore kvstore archiveName
    This step took a very long time (maybe its normal).

  • I monitored this using
    ./splunk show shcluster-status

  • The backupRestoreStatus finally moved to ready :

This member:
backupRestoreStatus : Ready
date : Fri Nov 29 13:34:12 2019
dateSec : 1575034452.206
disabled : 0
guid : 0C76D3C2-F11A-47FB-A705-3ECBC0CCE929
oplogEndTimestamp : Fri Nov 29 13:34:05 2019
oplogEndTimestampSec : 1575034445
oplogStartTimestamp : Fri Nov 29 10:11:49 2019
oplogStartTimestampSec : 1575022309
port : 8191
replicaSet : splunkrs
replicationStatus : KV store captain
standalone : 0
status : ready

Enabled KV store members:
guid : 0C76D3C2-F11A-47FB-A705-3ECBC0CCE929
hostAndPort : sh01:8191

KV store members:
configVersion : 1
electionDate : Fri Nov 29 13:24:26 2019
electionDateSec : 1575033866
hostAndPort : spplsh01:8191
optimeDate : Fri Nov 29 13:34:05 2019
optimeDateSec : 1575034445
replicationStatus : KV store captain
uptime : 608

But even if the kvstore status is all ok, when I search for data in the kvstores these are empty (even if there are lot of files in the mongo directory).
As this step is not ok, of course, I cannot go further trying to sync with another search head.

Has anyone already tried to do this ? maybe using another method ? for next steps, do I need to do the same on all SH of cluster or will the kvstores replicate automaticaly ?

Thanks in advance.

The used Splunk version is 7.3.2

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In the last month, the Splunk Threat Research Team (STRT) has had 1 release of new security content via the ...

There's No Place Like Chrome and the Splunk Platform

Watch On DemandMalware. Risky Extensions. Data Exfiltration. End-users are increasingly reliant on browsers to ...

The Great Resilience Quest: 5th Leaderboard Update

The fifth leaderboard update for The Great Resilience Quest is out >> 🏆 Check out the ...