Deployment Architecture

Moving Cluster to new subnet

dkeck
Influencer

Hi All,

I was wondering if there are any documentation are best practices for moving an indexer cluster to a new subnet.

I already checked conf files for IP addresses instead of DNS names, e.g. server.conf for master_URI

I am just wondering how the cluster will react to the change. I remember when I add new cluster peers I always entered the DNS name, but within the internal communication inside the cluster I have the feeling splunk is using the IP. Same with distributed search, when I check it on the SH (Index Cluster SH) the master is providing the IP not the DNS name.

Otherwise I would assume it like an upgrade?
1. Put Master in maintenance mode
2. Take peer offline, Change subnet
3. Start it up again
4. move forward with next peer on the same side

Or do I have to change all cluster peers on the same side at once?

So I would appreciate any help or hint on this.

PS: I assume that all firewall rules have been changed to allow communication between old and new subnet, since I am not the one doing this.

Thank you

David

0 Karma
1 Solution

esix_splunk
Splunk Employee
Splunk Employee

There's a few what ifs here.. Are you moving peers one/two at a time or lift and shift everything at once.

 

The lift and shift is the easiest in regards to moving subnets. Gracefully shutdown all your Splunk instances, move them to the new network, configure the OS, reconfigure Splunk via config files (USE DNS THIS TIME), test connectivity, bring everything back up... Shouldn't be any issue.

 

If you have to move servers one/two at a time, then it can be a bit more difficult as you need to validate IP and port connectivity across you network (never assume its going to work :>).  Before moving anything, I would go through your Splunk config files and find all references to IP and change them to fqdn's. Then make sure you have DNS correct, and if you can't update DNS, configure your /etc/hosts file to be a substitute for these hosts until you can.

From there, you can move your hosts incrementally. 

 

Otherwise your process is accurate and will work. 

View solution in original post

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

There's a few what ifs here.. Are you moving peers one/two at a time or lift and shift everything at once.

 

The lift and shift is the easiest in regards to moving subnets. Gracefully shutdown all your Splunk instances, move them to the new network, configure the OS, reconfigure Splunk via config files (USE DNS THIS TIME), test connectivity, bring everything back up... Shouldn't be any issue.

 

If you have to move servers one/two at a time, then it can be a bit more difficult as you need to validate IP and port connectivity across you network (never assume its going to work :>).  Before moving anything, I would go through your Splunk config files and find all references to IP and change them to fqdn's. Then make sure you have DNS correct, and if you can't update DNS, configure your /etc/hosts file to be a substitute for these hosts until you can.

From there, you can move your hosts incrementally. 

 

Otherwise your process is accurate and will work. 

0 Karma

dkeck
Influencer

Thank you, I will test this as you discribed 🙂

0 Karma
Get Updates on the Splunk Community!

How I Instrumented a Rust Application Without Knowing Rust

As a technical writer, I often have to edit or create code snippets for Splunk's distributions of ...

Splunk Community Platform Survey

Hey Splunk Community, Starting today, the community platform may prompt you to participate in a survey. The ...

Observability Highlights | November 2022 Newsletter

 November 2022Observability CloudEnd Of Support Extension for SignalFx Smart AgentSplunk is extending the End ...