PART2 How to migrate a distributed, clustered Splunk (9.0.4.1) deployment from OS RHEL 7 to new servers RHEL 9?

[Glasses2]

In some cases, I have Splunk installed on server builds where /opt/splunk is a mount point and separate disk/LVM.  That is /opt/splunk dir is on a different disk than the OS.

Would anyone know if there are any problems, detaching the LVM from a RHEL 7 box and then attaching the LVM to a RHEL 9 box?

Thank you

[gopu]

Since this is a distributed environment. I think the best approach here would be as follow:

1. Migrare your management nodes first like Cluster manager, SHC Deployer, Deployment Server, etc.
2. Migrate the Search tier
3. Migrate Indexers

Now the question is how do you achieve this with less to no downtime? 

For CM:

• Setup new RHEL 9 machines
• Backup 
• Setup Splunk on the new machine
• Copy/Merge "/system/local" and "etc/manager-apps" directory to the new machine 
• Restart Splunk 
• Update the DNS or IP. Easier way would be to deattach the DNS/IP of older CM and attach it to the new CM.
• Once the DNS/IP is update, your old indexers should start reporting to the new CM

For Indexers:

• Setup new RHEL 9 machines
• Now for example if you have 6 IDXs in your old cluster, then spawn 6 new machines in RHEL version
• Setup Splunk on the new machines 
• Ensure all the machines are part of one cluster(i.e older indexers and new both)
• Start decommissioning  old indexer one at a time. This will migrate the copy to the reminder IDXs: https://docs.splunk.com/Documentation/Splunk/9.2.1/Indexer/Takeapeeroffline
• Rebalnce the data if needed.
• This way you won't have any downtime on data ingestion or searching.

For SH 

• Setup new RHEL 9 machine
• Backup 
• Setup Splunk on the new machine
• Copy/Merge "/system/local" and "etc/apps" directory to the new machine 
• Restart Splunk 
• Update the DNS or IP. Easier way would be to deattach the DNS/IP of older SH and attach it to the new SH. (Optional)
• Ensure you are able to search the data as a part of sanity checks

Another approach is mentioned in this Splunk Docs which will be tedious and would require downtime and lot of manual work: 

Please let me know if you have any concerns. 

[Glasses2]

I think all the suggestions that spawned from my original question are worth considering in the context of the individual environments a user might have.  For instance, I was not able to spin up a duplicate SHC or IDXC and thus chose a different option.    However, I suggest that new threads be created for each solution, with a detailed explanation.  This thread is getting a bit long.

[iankitkumar]

hi @Glasses2 , did you found a way for this? even I have the same scenario for the migration

[Glasses2]

@iankitkumar 

Yes I did get it to work. 
But depends on how the disks and mount points are setup on the existing host. 
If the OS disk and Splunk disk are separate with distinct Volume Groups then a disk swap should work.  

For example, on a linux host where sda is the OS disk and sdb is the Splunk disk with mountpoint /opt/splunk.  
Here are the steps.   Note: I install with .tgz files not rpm.  Of course test in your dev environment 1st!!

1 install same version of Splunk on the new host with new Linux OS version (I use same username/pwd that I used to install Splunk on the existing host)

2 enable boot-start with systemd managed and verify install runs correctly

3 shut down the old and new hosts

4 swap the disk

5 rename/reIP and open required firewalld ports (if needed for your environment) to verify its working.

IF you cannot do a disk swap (because the host is not configured to do so) you can always tar up /opt/splunk and untar over /opt/splunk on the new host (on top of the fresh install of splunk (same version)). 

Documentation describes this method >>>  Migrate a Splunk Enterprise instance from one physical machine to another - Splunk Documentation

• Your Splunk Enterprise installation is on an operating system that either your organization or Splunk no longer supports, and you want to move it to an operating system that does have support.
  
  
  

Good Luck!

[PickleRick]

There are other similar threads on Answers - use the search functionality (yes, I know it can sometimes return results in a strange order).

The short and imprecise answer is - it should work but with caveats.

Longer answer is - since you probably have your Splunk instance installed by means of RPM package (which is a good thing on its own) just moving orphaned package contents without the package itself might be troublesome for later maintenance. So you might want to install an "empty" package first and then remove /opt/splunk and mount your LV in this place.

You also have to take care of creating service files for auto-starting the daemon.

And you have to make sure you move any additional content needed by your configuration (for example - certificate/key files if they are stored outside of your /opt/splunk directory).

EDIT: Of course it works assuming you want to replace an operating system "in-place" (retaining the same IP and hostname). If you want to move your instance to a host with a different name and IP things start getting messy.

[isoutamo]

I suggest to do it just opposite order. First mount your old splunk installation on /opt/splunk and after that install splunk from rpm to update rpm registry and ensure that correct files are in place. You should also use same splunk rpm version than you have on old node. After you have verify that it works then it's time to update.

[PickleRick]

Overwriting existing files with rpm -i  is not a very good idea. Especially if someone thought it smart to - for example - edit system/default files (as far as I can see, the splunk package doesn't have any files marked as config). Or tweaked log.cfg. Or...

[isoutamo]

With splunk updated I must disagree this. There are clear instructions how you should/must update your configurations etc. (don't update any default files etc.). It's better to get those issues fixed before you start production in a new environment or you will found those on 1st real update.

[PickleRick]

Yes, of course you should not do some things. It doesn't mean though that they are not done. Also, default was just one of the possible cases. Another is mentioned by me log.cfg. Yet another could be - manual fixing some errors within sendemail.py (yes, I had to do something like this a few versions back at one customer's site).

Reality can be complicated and "rpm, then sync files" lets you do the process in the controlled manner whereas "install files with rpm over existing directory contents" does not.

So I'm sticking with my order of doing this.