How to migrate a distributed, clustered Splunk (9.0.4.1) deployment from OS RHEL 7 to new servers RHEL 9?

[Glasses2]

I have a distributed deployment at version 9.0.4.1

Everything in running on RHEL 7 and the system/server team does not want to do in place upgrades to RHEL 9.  I have been tasked to migrate each node to a new replacement server (which will be renamed / IP- addressed to match the existing).   From what I have read this is possible, but I have a few questions.

Lets consider I start with standalone nodes, like a SHC-deployer, Monitoring Console, License Manager...
These are the general steps I have gathered

1 Install Splunk (same version) on the new server

2 Stop Splunk on the old server 

3 Copy old configs to new server ?? <<< which configs? is there a check list documented somewhere

4 Start new Splunk server and verify 

I could go thru each directory copying configs, but any advice to expedite this step is appreciated.

Thank you

[siemless]

Our team in is the same boat, we have to move our cluster off of RHEL7 to RHEL8. We also have to use the same IP addresses of the current peers to "mimic" the old instance to the new instance.

Since we have a multi-site cluster, we are going to take a peer from site1 down using splunk offline, then rsync the entire /opt/splunk dir to the new VM, ensure all network settings match and rejoin it to the cluster. Rinse and Repeat..

From the splunk docs, I'm confused on which step needs to happen first. 

@Glasses2 Which step did you perform first, did you install splunk rpm package on the new server? Or did you rsync the old instance to the new server first and then install splunk?

-siemless

[Glasses2]

@siemless 
This may be best to discuss in the Slack Users but I could not find you so I will respond here.

In some cases, I have boxes where the /opt/splunk dir is mounted to a separate drive w/ mount point /opt/splunk.  In that case you can just swap the disk, I learned this method from AWS support.  But that takes preplanning.

In some cases, I have boxes that are jacked up, either volume issues across multiple disks or  just not setup to swap disks.  In that case you can use the Splunk docs >>> https://docs.splunk.com/Documentation/Splunk/9.2.1/Installation/MigrateaSplunkinstance

I argued with Splunk about the documentation steps but they claim the steps are correct, although I still believe confusing. 

FWIW this is what I did...

1 >Create a new host with new OS (in my case I  rename /re-IP to the original afterward).
2 > Install the same version of Splunk on new host (I used a .tar), set systemd, set same admin pwd, then stop Splunkd, maybe test a restart and reboot, to verify.
3 > Stop Splunkd on old host, tar up /opt/splunk, copy over the old.tar to new box, untar over the new install, then start Splunkd.

That worked for me, and going fwd all new hosts will be configured for the disk-swappable process.
Good luck

[PickleRick]

One caveat with migrating filesystems directly between different instances of OS - it's relatively unlikely (especially with clean system installation) but as the file/directory ownership in the filesystem is set with UID/GID only, you might find yourself in a situation where UID/GID values of new/old system don't match. So - for example your splunk:splunk user might match 1002:1002 in your old Linux instance but your new one might be mapping splunk:splunk to 1005:1004 and 1002:1002 could be used by some interactive user.

So you might want to be doubly cautious where moving such filesystems between separate OS-es.

As far as I remember if packing/unpacking with tar ownership is by default preserved with username/groupname and used if such username/groupname are found in the OS when unpacking (unless you explicitly tell tar to just use numeric IDs).

[isoutamo]

Please create a new question as this one is already solved and you seems to have some different requirements.

[isoutamo]

Hi
My old answer for this https://community.splunk.com/t5/Splunk-Enterprise/Migration-of-Splunk-to-different-server-same-platf...
r. Ismo

[PickleRick]

Well... there are two possible approaches to migration of such environment.

First is as you want to do it - swap "one for one" leaving the same addresses, names and so on. You might get away with moving whole splunk installation from one server to another and pretending nothing changed but that might be tricky depending on your data layout and - you don't have much room for error - you replace the machine and it must be working perfectly OK. Otherwise it's very hard to diagnose/fix.

Another way, at least with some components (clustered indexers, clustered search heads, possibly HFs) would be to deploy new component, add it to environment, migrate data if applicable, decomission old one.

[Glasses2]

Thank you for the reply.
RE: "swap" method, yeah I thought about that and also share your apprehension.

RE: " deploy new component", yeah I agree with that method for idxc peers and shc members...

But check this out... please LMK what you think
Per Splunk docs >>> docs.splunk.com/Documentation/Splunk/9.2.0/Installation/MigrateaSplunkinstance

"Migrate a Splunk Enterprise instance from one physical machine to another"

"When to migrate"

"Your Splunk Enterprise installation is on an operating system that either your organization or Splunk no longer supports, and you want to move it to an operating system that does have support."

"How to migrate"
The Steps say >>>

1. Stop Splunk Enterprise services on the host from which you want to migrate.
2. Copy the entire contents of the $SPLUNK_HOME directory from the old host to the new host. Copying this directory also copies the mongo subdirectory.
3. Install Splunk Enterprise on the new host.

The way I read this is...
1) Stop Splunk on the old box
2) tar up the /opt/splunk on the old box e.g. > tar -cjvf $(hostname)_splunk-og.tar.bz2 --exclude=./var/* --exclude=./$(hostname)*bz2 ./

3) move and untar the .bz2 file on the new box in /opt  e.g. > tar -xjvf <hostname>_splunk-og.tar.bz2 -C /opt/splunk
4 ) install a clean copy (downloaded from Splunk) of the same version of Splunk on top of the old copies

Apparently someone that documented this believes this is the way to go... What do you think?

RE: my --exclude=./var/* that is for boxes that don't contain indexed data
RE: my --exclude=./$(hostname)*bz2./  this is because I am running the tar from /opt/splunk dir

Thank you

[PickleRick · Accepted solution]

Close but.

Depending on your environment, some components simply don't have to be replaced in place (for example HFs running modular inputs; caveat - network traffic and inputs state; also cluster manager does not _store_ any state locally - it builds up its in-memory database from what it queries from cluster peers).

If you want to do the "raze and replace", you'd rather install a fresh Splunk package and then unpack your archive (generally you want the package so that package manager's database is consistent with what's on the disk and you can still upgrade the softwarde properly).

$SPLUNK_HOME/var doesn't contain only indexed data. It contains fishbucket, kvstore, possibly state of modular inputs... You might want to exclude var/log though.

[Glasses2]

Ok thank you for the clarity...

I think someone should revise those steps then, its ambiguous.

[isoutamo]

Another old post to migrate individual splunk linux box into new one https://community.splunk.com/t5/Installation/How-to-migrate-indexes-to-new-indexer-instance/m-p/5280...