How to migrate a search head cluster to standalone search head?

[Chiranjeev88]

Hi,

i have to scale down my search head cluster to a standalone one but there is no documentation anywhere,

is it possible ?,what steps should i perform ?

[richgalloway · Accepted solution]

This is a rare event so it's unlikely to get documented.  Here's how I would do it.

1) Stand up a new SH and connect it to the indexer(s).

2) Copy app from the SHC deployer's shcluster directory to the new SH's apps directory.

3) Remove any [shclustering] stanzas in apps on the new SH.

4) Restart the new SH

5) Shut down and dismantle the SHC.

6) Remove [shclustering] stanzas from the indexer(s).  Restart indexer(s).

7) Update the Monitoring Console

[isoutamo]

Hi

one remark. When you have a SHC which has run some time, then there is a lot of modification which are not in deployer (unless there is strict change management implemented). For that reason I like to use any of SHC nodes as a source instead of deployer. 

Basically you should copy /opt/splunk/etc/{apps,users} from SHC node to the new sh node. Then install same splunk version than you have in SHC to that host. After that you have correct and unmodified default versions on place. 

If you want unique splunk.secret then remove old one before start or replace it with new one as docs instructed.

r. Ismo

[richgalloway]

Excellent point about local configs, @isoutamo .  Also, I forgot about user directories.

[Chiranjeev88]

Thanks for the detailed steps,could you tell what to do for kv store as well when migrating back from cluster to standalone @richgalloway / @isoutamo 

[richgalloway]

Backup the KVstore from the KVstore captain (not necessarily the same as SHC captain) and restore on the new SH.  Backup/restore steps are in https://docs.splunk.com/Documentation/Splunk/latest/Admin/BackupKVstore

[isoutamo]

You don’t need all kvstore collections. To copy only those which you are needing, you could use this app https://splunkbase.splunk.com/app/5328.

That can used also to do daily based kvstore backups.