Can we clone a splunk instance of Searchhead and indexer and add the new VM to the cluster

[bapun18]

Hi ,
We have a cluster of 3 searchheads and 3 indexers 2+1 primary and DR setup for both indexers and searchhead. If a DR indexer and a searchheads got corrupted, instead of  creating a new VM and install fresh splunk on the new VM and add it to the searchhed and indexer cluster is there a chance we can clone the existing searchhead and indexer VM to the new searchhead and indexer VM, and make it join the cluster.

[isoutamo]

As it has said, you basically could do it, but in reality I don't encourage you to do it.

As @livehybrid said your main issue will be to keep your clone sync with all changes what are happening on SHC member and also on indexers. In reality this means that you must have almost online sync to keep your clone enough fresh to use it successfully when time comes.

It's much better to add additional nodes to your DR site if you really need to have it up and running if master goes down and also your DR site's node is down.

When you are using Splunk's multisite cluster option you actually should have enough indexers on your primary and secondary sites. And if you are running this e.g. on AWS or Azure then just utilize those AZs to make your multisite environment. If you need to do this over regions / continents then I think that only reasonable way is use this multisite option. But then you cannot use SmartStore as it needs that all buckets are in same region.

On SHC side you need to ensure that you have enough members to get captain election works after breakup.

But to give you a better answer we must know what is the threat for you are planning to prepare.

r. Ismo

[gcusello · Accepted solution]

Hi @bapun18 ,

yes you can, but you should create a documented procedure containing all the steps to do when adding the new node, in particolar:

• change hostname and IP address in the server,
• change hostname in server.conf and in inputs.conf,
• change encrypted passwords,
• change the pointings to the clusters members (for SH).

Maybe it could be easier having a silent copy of the servers to start if there's a corruption.

Even if I don't imagine which kind of corruption you are speaking of.

Ciao.

Giuseppe

[livehybrid]

Hi @bapun18 

I think this ultimately comes down to how you clone them, and when. If you clone a corrupted installation then theres a good chance you will end up with a corrupted clone of it. If you take a clone and have it waiting offline, then it could be hugely out of the date.

The success of this approach really depends on your Splunk architecture, and how your configuration and data is managed.

Please let me know how you get on and consider accepting this answer or adding karma this answer if it has helped.
Regards

Will