Deployment Architecture

How to migrate KV store data from a search head standalone to a search head cluster ?

mgoueroupro
New Member

Hello,

I have a standalone search head with KVstores.
I want to migrate the KVstores to a search head cluster without, if possible, exporting all data (in csv or other format) and importing them again as it represents a large quantity of data (2-3GB) and many collections.

What I tryed :

  • backup the kvstores from the standalone server using
    ./splunk backup kvstore

  • Set the replication factor to 1 on one search head of the new cluster

  • Clean kvstore db on this search head :
    ./splunk clean kvstore --local
    ./splunk clean kvstore --cluster

  • Restore on the clustered SH the backuped kvstore from archive
    ./splunk restore kvstore archiveName
    This step took a very long time (maybe its normal).

  • I monitored this using
    ./splunk show shcluster-status

  • The backupRestoreStatus finally moved to ready :

This member:
backupRestoreStatus : Ready
date : Fri Nov 29 13:34:12 2019
dateSec : 1575034452.206
disabled : 0
guid : 0C76D3C2-F11A-47FB-A705-3ECBC0CCE929
oplogEndTimestamp : Fri Nov 29 13:34:05 2019
oplogEndTimestampSec : 1575034445
oplogStartTimestamp : Fri Nov 29 10:11:49 2019
oplogStartTimestampSec : 1575022309
port : 8191
replicaSet : splunkrs
replicationStatus : KV store captain
standalone : 0
status : ready

Enabled KV store members:
spplsh01:8191
guid : 0C76D3C2-F11A-47FB-A705-3ECBC0CCE929
hostAndPort : sh01:8191

KV store members:
spplsh01:8191
configVersion : 1
electionDate : Fri Nov 29 13:24:26 2019
electionDateSec : 1575033866
hostAndPort : spplsh01:8191
optimeDate : Fri Nov 29 13:34:05 2019
optimeDateSec : 1575034445
replicationStatus : KV store captain
uptime : 608

But even if the kvstore status is all ok, when I search for data in the kvstores these are empty (even if there are lot of files in the mongo directory).
As this step is not ok, of course, I cannot go further trying to sync with another search head.

Has anyone already tried to do this ? maybe using another method ? for next steps, do I need to do the same on all SH of cluster or will the kvstores replicate automaticaly ?

Thanks in advance.

The used Splunk version is 7.3.2

0 Karma
Get Updates on the Splunk Community!

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...

Introducing Splunk Enterprise 9.2

WATCH HERE! Watch this Tech Talk to learn about the latest features and enhancements shipped in the new Splunk ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...