Deployment Architecture

Moving Cluster to new subnet

dkeck
Influencer

Hi All,

I was wondering if there are any documentation are best practices for moving an indexer cluster to a new subnet.

I already checked conf files for IP addresses instead of DNS names, e.g. server.conf for master_URI

I am just wondering how the cluster will react to the change. I remember when I add new cluster peers I always entered the DNS name, but within the internal communication inside the cluster I have the feeling splunk is using the IP. Same with distributed search, when I check it on the SH (Index Cluster SH) the master is providing the IP not the DNS name.

Otherwise I would assume it like an upgrade?
1. Put Master in maintenance mode
2. Take peer offline, Change subnet
3. Start it up again
4. move forward with next peer on the same side

Or do I have to change all cluster peers on the same side at once?

So I would appreciate any help or hint on this.

PS: I assume that all firewall rules have been changed to allow communication between old and new subnet, since I am not the one doing this.

Thank you

David

0 Karma
1 Solution

esix_splunk
Splunk Employee
Splunk Employee

There's a few what ifs here.. Are you moving peers one/two at a time or lift and shift everything at once.

 

The lift and shift is the easiest in regards to moving subnets. Gracefully shutdown all your Splunk instances, move them to the new network, configure the OS, reconfigure Splunk via config files (USE DNS THIS TIME), test connectivity, bring everything back up... Shouldn't be any issue.

 

If you have to move servers one/two at a time, then it can be a bit more difficult as you need to validate IP and port connectivity across you network (never assume its going to work :>).  Before moving anything, I would go through your Splunk config files and find all references to IP and change them to fqdn's. Then make sure you have DNS correct, and if you can't update DNS, configure your /etc/hosts file to be a substitute for these hosts until you can.

From there, you can move your hosts incrementally. 

 

Otherwise your process is accurate and will work. 

View solution in original post

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

There's a few what ifs here.. Are you moving peers one/two at a time or lift and shift everything at once.

 

The lift and shift is the easiest in regards to moving subnets. Gracefully shutdown all your Splunk instances, move them to the new network, configure the OS, reconfigure Splunk via config files (USE DNS THIS TIME), test connectivity, bring everything back up... Shouldn't be any issue.

 

If you have to move servers one/two at a time, then it can be a bit more difficult as you need to validate IP and port connectivity across you network (never assume its going to work :>).  Before moving anything, I would go through your Splunk config files and find all references to IP and change them to fqdn's. Then make sure you have DNS correct, and if you can't update DNS, configure your /etc/hosts file to be a substitute for these hosts until you can.

From there, you can move your hosts incrementally. 

 

Otherwise your process is accurate and will work. 

0 Karma

dkeck
Influencer

Thank you, I will test this as you discribed 🙂

0 Karma
Get Updates on the Splunk Community!

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...