Cluster - Mix of RHEL 8 and RHEL 9 Problematic?

[VeloPunk]

I'm on the server / infrastructure team at my organization. There is a dedicated Splunk team, and they want to replace some RHEL7 Splunk servers with RHEL 8. RHEL 8 is already near the end of its lifecycle, and I'd rather provide them with RHEL 9, which is now our standard build. The fact that they still use RHEL 7 servers gives you some sense of how long it takes them to move their application to a new(ish) OS. They are insistent that we deploy them RHEL 8 servers so they are "all the same." I want to encourage them to move forward and have a platform that will be fully supported for several  years to come. Is having some servers on RHEL 8 and some on RHEL 9 for a period of time an actual problem? They use version 9.1.2. I found this document:

https://docs.splunk.com/Documentation/Splunk/9.1.2/Installation/Systemrequirements

It lists support both for x86_64 kernels 4.x (rhel 😎 and 5.x (rhel 9). It doesn't elaborate any further. 

I know that for various reasons we'd want to eventually have all servers on the same OS version; I'm just wondering if having RHEL 8 and RHEL 9 coexist for a limited period presents an actual problem. I'd appreciate your thoughts. 

Daniel

[ZaheerCG]

All, just need some advice. We have a customer that we are migrating across different cloud providers. Their current Splunk cluster is running on Ubuntu 20.04 (which goes end of life 31st of May). We want to add new nodes in the new cloud provider running on RHEL 9.x and extend the existing Splunk cluster. So for a shortwhile we will have a mix of Ubuntu and RHEL nodes running together in the same cluster. Splunk have said this is doable but not something they can guarantee as they are not responsible for the OS running on the cluster nodes.

Below is the below migration plan we are proposing to the customer, once we get approval we will deploy a PoC to test the migration approach:

1. Ensure there is low latency and sufficient bandwidth between the two cloud providers
2. Deploy new RHEL nodes in new Cloud provider with the same version of Splunk
3. Add the RHEL nodes to the existing Ubuntu cluster and let the cluster synchronize the data to the new RHEL nodes
4. Following successful data synchronization and testing, the master cluster role will be transferred to one of the RHEL nodes
5. Finally, after a period of co-existence and validation, the existing Ubuntu nodes will be removed from the cluster (indexers and Search heads).

Any help or guidance appreciated.

[isoutamo]

As I earlier said this should be doable. Just try to keep the mixed mode time as short as possible. Of course you must have enough wide and fast connections between your environments, but currently I don’t think that this is an issue.

[PickleRick]

As it was already stated - the OS differences shouldn't be that problematic from the technical point of view. It can add some maintenance overhead because you have to maintain different package types, maybe different service launch methods, but generally, the software should work.

It will be unsupported though.

But the main issue with such setup is that you'll have a relatively spread out cluster. Here your main issue will probably be latency across your envionment. https://docs.splunk.com/Documentation/Splunk/latest/Capacity/Referencehardware#:~:text=A%20Splunk%20....

[VeloPunk]

I appreciate everyone's input on this! I ended up deploying RHEL 8 servers for now. I will nudge them towards RHEL 9 when they are ready to upgrade the version of their Splunk cluster. 

Thanks!

Daniel

[PickleRick]

There are different factors at play here.

Of course Splunk must be supported on the systems used. That's obvious.

If you use clusters, the docs state that the same system/version is required for all nodes of the cluster. That is a bit vague and there has been a lot of discussion about what it actually means but just to be on the safe side you should stick to the same release across all nodes of an indexer cluster or search head cluster.

There is of course the general issue of maintainability but that's a double-edged sword. Uniform environment is of course easier to maintain but there's less work if you don't have to upgrade your systems soon. So it's your call.

There is no requirement that whole environment must be using the same OS release (and there can't be given thay you can have separate search-heads (or even SHCs) operated by - for example - different divisions of the company searching against the same indexers. Or you can have many different HFs doing different modular inputs. Some of them even could be windows-based. I managed environments where - for example - some servers were CentOS and some were SUSE and nothing blew up 😉

So as long as you're not mixing systems across a cluster you should be fine.

[isoutamo]

Even in cluster you could use different versions if you haven’t any other options. But try to limit this time as short as possible. In practice it means the time which you need to update all nodes in cluster to same version. Same is valid for OS and Splunk versions too.

I think that easiest way this can do by adding new servers with a new OS version BUT the same splunk version than you have in your cluster’s other nodes. 
Here is an old post how this can do https://community.splunk.com/t5/Splunk-Enterprise/Migration-of-Splunk-to-different-server-same-platf.... This was a play how I migrate distributed splunk environment into a new service provider into newer OS without service breaks.

The migration took couple of weeks but less than month.

[PickleRick]

Well... there are two different views on that 😉

Technically you can do several things which aren't officially supported and which - while they do work - can get you into a "sorry, that's an unsupported setup" situation.

But yes - if you have a cluster on RHEL8 and want to upgrade/migrate it to RHEL9, short of turning the whole system down and upgrading all servers at once you have no other option than to have some servers on one system, some or another. But I'd definitely go for minimizing time the cluster was in that state.

[isoutamo]

I totally agree this, but when you must keep business up and running and there is this things we called physic, we haven’t any other option than wait. Here is old story about it https://web.mit.edu/jemorris/humor/500-miles

[gcusello]

Hi @VeloPunk ,

for my knowledge, it shouldn't be a probem, the only mandatory requirement is that Splunk must be the same.

Then RHEL8 or 9 should be the same, obviously it should be better having the same version, but for a transient period, they should live together.

To have an official answer, open a case to Splunk Support or ask yo your Splunk Sales Engineer.

Ciao.

Giuseppe

[isoutamo]

Splunk should works on both RHEL 8 or 9, but with 9 there are some additional steps which one must do before splunk can installed there. RHEL 9 have cgroups v2 as default and that version of Splunk support only version 1. There are probably also some security changes which must notice before splunk works correctly. If I recall right at least some of those are already on 8 and maybe some more in 9?