We are migrating SHC members from old to new datacenter. There are total 3 members as a part of SHC.
Please tell us which is the best approach to follow.
1. Add 1 new node to the SHC, have the things replicated and then decommission the old node. Repeat this step untill all the nodes are migrated with replicated bundles.
2. Stop Old SHC members, take backup of SHC run time bundles. Install 3 new members and push configs with Deployer first and later restore the old SHC run time bundles.
Appreciate your help on this
Today I tried the same but has fallen in some issue. Our Replication factor of SHC is 3.
So we added a new Node, added it to SHC and CM. Removed one OLD Node which makes 2 OLD and 1 NEW Node in SHC.
Searches and Reports,alerts have been replicated perfectly but Apps have been migrated partially. We don't know whats the reason behind this. Could you please help.
Yes SHC status and KV store status is successful and i am no longer seeing the old node but new node.
Regarding Shcluster-bundle command , i didn't apply as document suggests that new node automatically takes the configuration from Deployer, we don't need to push anything specifically.
My biggest fear is if i apply shcluster-bundle command it will overwrite the existing settings saved-searches/dashboards/alerts on SHC members since I last push. shccluster always contains old data but not the replicated one which was exchanged between SHC members.
Please correct my understanding.
Apply shcluster-bundle is a normal way to manage SHC. You could read more about it's behaviour for files which are only in SHC nodes but not in Deployer.
Default mode is not overwrite those in SHC nodes' local folders. You can change this behaviour if you want based on above instructions.
Thanks. As per the documentation the default push mode is merge_to_default.
splunk apply shcluster-bundle -target <URI>:<management_port> -auth <username>:<password>
So the above command should not overwrite the existing configurations. Let me try with that.
Yes you are right.
So finally it worked when i pushed the bundle and surprisingly when i added the other two nodes it was not required to push the bundles again.
Thank you for the help. Cheers !
Thanks so we will follow the same approach.
one last question
1. How do we know that bundles are replicated so that we can proceed to other nodes.
2. Did you follow the same approach with Index clustering.
Wait couple of minutes and then check that both shcluster and kvstore status are ok.
In our case we first add all new nodes and then removed old ones after some time.