Deployment Architecture

How to migrate KV store data from a search head standalone to a search head cluster ?

mgoueroupro
New Member

Hello,

I have a standalone search head with KVstores.
I want to migrate the KVstores to a search head cluster without, if possible, exporting all data (in csv or other format) and importing them again as it represents a large quantity of data (2-3GB) and many collections.

What I tryed :

  • backup the kvstores from the standalone server using
    ./splunk backup kvstore

  • Set the replication factor to 1 on one search head of the new cluster

  • Clean kvstore db on this search head :
    ./splunk clean kvstore --local
    ./splunk clean kvstore --cluster

  • Restore on the clustered SH the backuped kvstore from archive
    ./splunk restore kvstore archiveName
    This step took a very long time (maybe its normal).

  • I monitored this using
    ./splunk show shcluster-status

  • The backupRestoreStatus finally moved to ready :

This member:
backupRestoreStatus : Ready
date : Fri Nov 29 13:34:12 2019
dateSec : 1575034452.206
disabled : 0
guid : 0C76D3C2-F11A-47FB-A705-3ECBC0CCE929
oplogEndTimestamp : Fri Nov 29 13:34:05 2019
oplogEndTimestampSec : 1575034445
oplogStartTimestamp : Fri Nov 29 10:11:49 2019
oplogStartTimestampSec : 1575022309
port : 8191
replicaSet : splunkrs
replicationStatus : KV store captain
standalone : 0
status : ready

Enabled KV store members:
spplsh01:8191
guid : 0C76D3C2-F11A-47FB-A705-3ECBC0CCE929
hostAndPort : sh01:8191

KV store members:
spplsh01:8191
configVersion : 1
electionDate : Fri Nov 29 13:24:26 2019
electionDateSec : 1575033866
hostAndPort : spplsh01:8191
optimeDate : Fri Nov 29 13:34:05 2019
optimeDateSec : 1575034445
replicationStatus : KV store captain
uptime : 608

But even if the kvstore status is all ok, when I search for data in the kvstores these are empty (even if there are lot of files in the mongo directory).
As this step is not ok, of course, I cannot go further trying to sync with another search head.

Has anyone already tried to do this ? maybe using another method ? for next steps, do I need to do the same on all SH of cluster or will the kvstores replicate automaticaly ?

Thanks in advance.

The used Splunk version is 7.3.2

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...