Running Cluster Masters for servers on Different RHEL version

[Abass42]

I am upgrading from RHEL 7 to RHEL 8 in light of end of support for Red Hat. We have a clustered environment. We have two sites per cluster for each the SH and Indexer Cluster. All splunk servers are on 9.2.0.1. 

 My question is, can we run a RHEL 8 Cluster Master and have a mixed environment of RH 8 and RH 7 servers within the cluster? I know there is a hierarchy for the servers, but i wasn't sure to what extent the OS affected the application. 

With the upgrade, I might have:

RHEL 8 Indexer Cluster Manager while Indexers themselves are on RHEL 7.

RHEL 8 SH cluster Manager while SH may be on RHEL 7. Depending on how the in-place upgrade goes, determines how many servers I upgrade at once. These are all Azure servers or VmWare servers. 

Would any search functionality for any of the search peers be affected by differing OS versions?

Thank you for any clarity. 

[PickleRick · Accepted solution]

You can find earlier discussions on this topic within Answers. It's... a bit tricky.

There are at least several different things at play here.

1. The technicalities - as Splunk brings with it most of the things it requires and generally uses just the "bare OS" level of what is provided by OS, there shouldn't be a problem with running different OS versions (even different distros - I run a combined CentOS/SuSE environment for some time). It should work.

2. The maintainability - as different distros and different releases have some different mechanisms (like different startup scripts, different ways of configuring the system and so on), a mixed environment is much more prone to errors and misconfiguration.

3. The supportability - the official docs say that all cluster members must run the same "OS and version". And here's where it gets really tricky - there is no single official explanation what this means so while technically it could just mean that all boxes must be Linux-based and they must be running 64-bit OS version (and that could really be the bare minimum to make the cluster work), it can also be understood as "all boxes must use the same Linux distro and they all must be running the same release".

So long story short - from the technical point of view it usually doesn't make much of a difference whether you're running RHEL9 across your whole environment or if some boxes are still at RHEL8 (if you already have a RHEL8 environment and want to migrate to RHEL9, you will have at some point a situation when some boxes are already migrated and some are not) but if you raise a support case and support finds out that you have a mixed setup, they might want to tell you "get your environment in order and align your OS versions".

[Abass42]

Thank you both for the prompt response. 

This is the impression I was under as well when speaking with our Splunk rep. Now that we are upgrading, the mis match of OS will only be for a week or two at most, but was wanting to confirm that the data availability/searchability wouldn't be affected. Thank you both for the resources and in depth answers given. 

[livehybrid]

Hi @Abass42 

From my experience you can run a Splunk cluster (indexer or search head) with a mix of RHEL 7 and RHEL 8 hosts, including having your cluster manager on RHEL 8 while some peers remain on RHEL 7, as long as all Splunk nodes are running the same supported Splunk version. 

The underlying OS version does not affect Splunk clustering compatibility, provided both OS versions are supported by the Splunk version in use, although it becomes much more complicated if mis-matching underlying OS e.g. Windows Vs Linux!

This is from a technical standpoint though, While mixed OS versions may be supported during migration periods, the recommended long-term state is to standardize all nodes in the cluster on the same, newer supported OS version. You run the risk of being in an un-supported state if you remain in a mixed version state.

OS differences alone should not impact search or cluster management functionality, Splunk communicates via supported network protocols, not OS-specific mechanisms. 

You have possibly already seen this but its worth reviewing https://help.splunk.com/en/splunk-enterprise/get-started/install-and-upgrade/9.2/plan-your-splunk-en... 

I would also recommend that after migrating all nodes to RHEL 8, to revalidate ulimits etc.

Avoid running mismatched Splunk software versions across cluster nodes/IDX where possible to avoid different performance across different nodes.

If you use custom scripts or apps, validate their dependencies (Python, OS libraries) for OS compatibility.

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[Abass42]

Thank you. I would set this as the solution, but i can only do one solution 😞

Thank you for your time.