Migration of Search head clustering

[dvohra]

Hi All,

We are migrating SHC members from old to new datacenter. There are total 3 members as a part of SHC.

Please tell us which is the best approach to follow.

1. Add 1 new node to the SHC, have the things replicated and then decommission the old node. Repeat this step untill all the nodes are migrated with replicated bundles.

2. Stop Old SHC members, take backup of SHC run time bundles. Install 3 new members and push configs with Deployer first and later restore the old SHC run time bundles.

Appreciate your help on this

@gcusello @somesoni2 

[gcusello]

Hi @dvohra,

Yes Option 1 it's correct.

I'd move before the Deployer and, after the first moving, I'd elect the new member as Captain.

Ciao.

Giuseppe

[dvohra]

Hi @gcusello @isoutamo 

Today I tried the same but has fallen in some issue. Our Replication factor of SHC is 3.

So we added a new Node, added it to SHC and CM. Removed one OLD Node which makes 2 OLD and 1 NEW Node in SHC.

Searches and Reports,alerts have been replicated perfectly but Apps have been migrated partially. We don't know whats the reason behind this. Could you please help.

Regards,

Devang

[isoutamo]

Did your captain show all members as ok (both shc and kvstore status)?
Have you try apply shcluster-bundle?
What your MC sows as SHC status?

[dvohra]

Yes SHC status and KV store status is successful and i am no longer seeing the old node but new node.

Regarding Shcluster-bundle command , i didn't apply as document suggests that new node automatically takes the configuration from Deployer, we don't need to push anything specifically.

[isoutamo · Accepted solution]

Please try apply shcluster-bundle. You need to test it anyhow for further operations.
Usually we first add all new nodes, then check that everything is working incl. apply + rolling restart. After everything is verified then we start to remove those old nodes and again test all before we said that it's working.

[dvohra]

My biggest fear is if i apply shcluster-bundle command it will overwrite the existing settings saved-searches/dashboards/alerts on SHC members since I last push. shccluster always contains old data but not the replicated one which was exchanged between SHC members.

Please correct my understanding. 

[isoutamo]

Apply shcluster-bundle is a normal way to manage SHC. You could read more about it's behaviour for files which are only in SHC nodes but not in Deployer.

https://docs.splunk.com/Documentation/Splunk/8.1.2/DistSearch/PropagateSHCconfigurationchanges

Default mode is not overwrite those in SHC nodes' local folders. You can change this behaviour if you want based on above instructions.

r. Ismo

[dvohra]

Thanks. As per the documentation the default push mode is merge_to_default.

splunk apply shcluster-bundle -target <URI>:<management_port>  -auth <username>:<password>

 So the above command should not overwrite the existing configurations. Let me try with that.

B.R

[isoutamo]

Yep. One comment on security point of view. Newer use that -auth <user>:<pass> on command line. Those can found from history and process list. Much better is leave those away and when splunk separately asks those then give them without fear that anyone can figure out those later on.
I haven't newer understood why those are put on examples on every command 😞
r. Ismo

[dvohra]

Yes you are right.

So finally it worked when i pushed the bundle and surprisingly when i added the other two nodes it was not required to push the bundles again.

Thank you for the help. Cheers !

[isoutamo]

Hi

we have done it successful with option one. One of it’s benefits is that it can do without service break.

R. Ismo

[dvohra]

Thanks so we will follow the same approach.

one last question 

1. How do we know that bundles are replicated so that we can proceed to other nodes. 

2. Did you follow the same approach with Index clustering.

[isoutamo]

Wait couple of minutes and then check that both shcluster and kvstore status are ok.

In our case we first add all new nodes and then removed old ones after some time.