CentOS migration to Ubuntu

[mikefg]

I am working on migrating from Centos 7 to Ubuntu 22. Single search head, indexer cluster (3 indexers), and a deployment server used just to manage clients (not Splunk servers).

For the SH and DS is it just a straightforward install same version Splunk on new Ubuntu server, copy config over, check permissions, and start it up (same IP, same DNS)?

For the IDX cluster, do a new CM first and copy config over or are there other things to consider?

What's a good process for the indexers (only 3). Can I build new indexers on Ubuntu, add them to the cluster, and then remove the CentOS servers as new Ubuntu servers are added all the while letting clustering handle the data management?

[richgalloway]

I think you have right idea on all counts.  Migrating the CM is similar to migrating a SH.  Do migrate the CM before the indexers.

[mikefg]

---

@richgalloway wrote:

I think you have right idea on all counts.  Migrating the CM is similar to migrating a SH.  Do migrate the CM before the indexers.

---

Working on this project. I have the new CM stood up on Ubuntu 22 and it has replaced the Centos 7 CM which is now offline. The Indexers are still on Centos 7. I see in the docs that the CM and indexers need to be the same OS. Is this true?

The cluster seems to be working fine so far and I'm working on the new Ubuntu indexers that will be added to the cluster.

Still safe to proceed or will I run into issues adding the Ubuntu indexers to the cluster?

Found under "Operating system requirements"
"All indexer cluster nodes (manager node, peer nodes, and search heads) must run on the same operating system and version."
System requirements and other deployment considerations for indexer clusters - Splunk Documentation

[PickleRick]

+1 to what @richgalloway wrote - the official requirements are a bit... imprecise here and noone really knows how to interpret them.

From my personal experience, it means that:

1) All cluster members should be running on the same operating systems - 100% Linux cluster or 100% windows cluster

2) All member should run on the same architecture (I don't remember if there are 32-bit versions available anymore but back when they were it might have mattered so you mustn't mix 32-bit and 64-bit; and of course don't try to add to the mix any ARMs if/when they become available)

3) As long as the cluster members are properly set up on each respective OS they should work but it is a good practice to keep things homogenous - it saves you on maintenance and troubleshooting. Also Splunk Support can reject cases if you have mixed environment especially if an issue is present on one OS and not showing on another.

[richgalloway]

The OS requirement is somewhat flexible to allow for OS upgrades, patches, etc.  In my mind, it means Linux vs Windows more than Ubuntu vs CentOS.  That said, every effort should be made to have the CM and indexers on the same release.

You should have no problems adding the Ubuntu indexers to the cluster.

[mikefg]

Thanks for clarifying. I have added the new Ubuntu servers to the cluster and will initiate the data rebalance today. 

Doing some reading on removing a peer and it looks like it's as simple as running the offline command on the peer, correct? Assuming the cluster is healthy, just run this wait for it to finish, and the server can be shut down.

   splunk offline --enforce-counts

[richgalloway]

You are correct.

[mikefg]

Thanks for responding. I'll proceed and see how it goes!

[isoutamo]

Hi

here is an old post about migrating distributed splunk environment. As long as you use Linux there shouldn’t issues with different os distroes under migration time. Just keep splunk version same on old and new nodes until you have done the migration. 

https://community.splunk.com/t5/Splunk-Enterprise/Migration-of-Splunk-to-different-server-same-platf...

r. Ismo

[mikefg]

This is great! Good steps to follow, thank you!