topic Re: Migration of Search head clustering in Deployment Architecture

Migration of Search head clustering

dvohra — Thu, 04 Mar 2021 15:16:37 GMT

Hi All,

We are migrating SHC members from old to new datacenter. There are total 3 members as a part of SHC.

Please tell us which is the best approach to follow.

1. Add 1 new node to the SHC, have the things replicated and then decommission the old node. Repeat this step untill all the nodes are migrated with replicated bundles.

2. Stop Old SHC members, take backup of SHC run time bundles. Install 3 new members and push configs with Deployer first and later restore the old SHC run time bundles.

Appreciate your help on this

@gcusello @somesoni2

Re: Migration of Search head clustering

isoutamo — Thu, 04 Mar 2021 15:44:54 GMT

we have done it successful with option one. One of it’s benefits is that it can do without service break.

R. Ismo

Re: Migration of Search head clustering

dvohra — Thu, 04 Mar 2021 15:56:38 GMT

Thanks so we will follow the same approach.

one last question

1. How do we know that bundles are replicated so that we can proceed to other nodes.

2. Did you follow the same approach with Index clustering.

Re: Migration of Search head clustering

isoutamo — Thu, 04 Mar 2021 19:13:06 GMT

Wait couple of minutes and then check that both shcluster and kvstore status are ok.

In our case we first add all new nodes and then removed old ones after some time.

Re: Migration of Search head clustering

gcusello — Fri, 05 Mar 2021 07:34:30 GMT

Hi @dvohra,

Yes Option 1 it's correct.

I'd move before the Deployer and, after the first moving, I'd elect the new member as Captain.

Ciao.

Giuseppe

Re: Migration of Search head clustering

dvohra — Tue, 09 Mar 2021 16:20:39 GMT

Hi @gcusello @isoutamo

Today I tried the same but has fallen in some issue. Our Replication factor of SHC is 3.

So we added a new Node, added it to SHC and CM. Removed one OLD Node which makes 2 OLD and 1 NEW Node in SHC.

Searches and Reports,alerts have been replicated perfectly but Apps have been migrated partially. We don't know whats the reason behind this. Could you please help.

Regards,

Devang

Re: Migration of Search head clustering

isoutamo — Tue, 09 Mar 2021 22:22:47 GMT

Did your captain show all members as ok (both shc and kvstore status)?
Have you try apply shcluster-bundle?
What your MC sows as SHC status?

Re: Migration of Search head clustering

dvohra — Wed, 10 Mar 2021 05:22:48 GMT

Yes SHC status and KV store status is successful and i am no longer seeing the old node but new node.

Regarding Shcluster-bundle command , i didn't apply as document suggests that new node automatically takes the configuration from Deployer, we don't need to push anything specifically.

Re: Migration of Search head clustering

isoutamo — Wed, 10 Mar 2021 06:48:27 GMT

Please try apply shcluster-bundle. You need to test it anyhow for further operations.
Usually we first add all new nodes, then check that everything is working incl. apply + rolling restart. After everything is verified then we start to remove those old nodes and again test all before we said that it's working.

Re: Migration of Search head clustering

dvohra — Wed, 10 Mar 2021 07:06:04 GMT

My biggest fear is if i apply shcluster-bundle command it will overwrite the existing settings saved-searches/dashboards/alerts on SHC members since I last push. shccluster always contains old data but not the replicated one which was exchanged between SHC members.

Please correct my understanding.

Re: Migration of Search head clustering

isoutamo — Wed, 10 Mar 2021 07:39:29 GMT

Apply shcluster-bundle is a normal way to manage SHC. You could read more about it's behaviour for files which are only in SHC nodes but not in Deployer.

https://docs.splunk.com/Documentation/Splunk/8.1.2/DistSearch/PropagateSHCconfigurationchanges

Default mode is not overwrite those in SHC nodes' local folders. You can change this behaviour if you want based on above instructions.

r. Ismo

Re: Migration of Search head clustering

dvohra — Wed, 10 Mar 2021 08:04:48 GMT

Thanks. As per the documentation the default push mode is merge_to_default.

splunk apply shcluster-bundle -target <URI>:<management_port>  -auth <username>:<password>

So the above command should not overwrite the existing configurations. Let me try with that.

B.R

Re: Migration of Search head clustering

isoutamo — Wed, 10 Mar 2021 08:20:25 GMT

Yep. One comment on security point of view. Newer use that -auth <user>:<pass> on command line. Those can found from history and process list. Much better is leave those away and when splunk separately asks those then give them without fear that anyone can figure out those later on.
I haven't newer understood why those are put on examples on every command 😞
r. Ismo

Re: Migration of Search head clustering

dvohra — Wed, 10 Mar 2021 17:52:44 GMT

Yes you are right.

So finally it worked when i pushed the bundle and surprisingly when i added the other two nodes it was not required to push the bundles again.

Thank you for the help. Cheers !