Best way to upgrade/migrate Splunk indexers to new hardware - more details included

[Bomo2023]

I currently have a Splunk cluster that looks like this:

Splunk	CentOS Version	Splunk Version
Master	7.5	7.0.0
Forwarder	7.5	Universal Forwarder 6.6.3
Search Head	6.5	7.0.0
Indexer 1	6.5	7.0.0
Indexer 2	6.5	7.0.0
Indexer 3	6.5	7.0.0
Indexer 4	6.5	7.0.0

I have 4 new servers that I want to build as indexers to replace the 4 indexers that I currently have (while moving from Splunk 7.0.0 to Splunk 8.x in the process).

My first plan was to build the 4 new indexers and then join them to the current indexer cluster so that existing data could replicate between them and then I could retire the old indexers from the cluster.

The problem with this idea is that my existing indexers are running CentOS 6.x and my new indexers will be running CentOS 7.x - as I understand it from reading documentation it is not possible to have different OS versions running in the same indexer cluster. As there is no easy way to upgrade from CentOS 6.x to CentOS 7.x, I'd rather avoid having to do this.

So my next idea was to build the new indexers as a separate cluster - so that I have one cluster of CentOS 6.x indexers and one cluster of CentOS 7.x indexers - then send new data only to the new cluster and let old data eventually age off on the old cluster at which point I can retire it. During this time I will have a Search Head that searches across both clusters, so that old and new data is searchable.

My question on that is:
Will my Search Head running on CentOS 6.x be able to search across both a CentOS 6.x indexer cluster and a CentOs 7.x indexer cluster?

Should I instead look at creating the second cluster of indexers and then manually migrating data between the old and new clusters? Although that seems like a harder process.

How would you approach this?

[PickleRick · Accepted solution]

Splunk version has nothing to do (except possibly the compatibility between one another) with OS version. There is no technical reason why indexers running on CentOS6 wouldn't work with indexers running on CentOS7.

Furthermore, it is sometimes possible to upgrade from CentOS6 to CentOS7 in place (did it myself on few servers) but the procedure is not straightforward and there can be hard to resolve problems down the road.

[Bomo2023]

I thought that all nodes needed to be on the same OS version because of this:

https://docs.splunk.com/Documentation/Splunk/8.2.2/Indexer/Systemrequirements

All indexer cluster nodes (manager node, peer nodes, and search heads) must run on the same operating system and version.

If the indexer cluster is integrated with a search head cluster, then the search head cluster instances, including the deployer, must run on the same operating system and version as the indexer cluster nodes.

---

Or is that not a hard and fast rule?

If it's the case that I can mix OS versions in an indexer cluster, in light of that, do you think it would be best to join the new indexers into the same cluster as the old indexers and let data replicate between them before retiring the old indexers? Or do you think it would be best to go with my original plan of dividing them into two separate clusters, send new data to the new cluster and search across both clusters until the old data ages off?

[somesoni2]

The search head must be at the same or a higher level than the search peers. (reference here), so you won't be able to search across both cluster if you're not planning to upgrade your SH first (assuming your old cluster is on v7 and new cluster on v8, upgrade your SH to v8 first). OS version shouldn't be a problem.

Having two cluster and retiring old cluster when time comes would be the easier approach but if the HW of both indexers are not same, it could lead to some performance deficiencies. If your retention period are not  very high, go with two cluster approach. 

Data migration is complex process. 

[Bomo2023]

I thought that all nodes needed to be on the same OS version because of this:

https://docs.splunk.com/Documentation/Splunk/8.2.2/Indexer/Systemrequirements

All indexer cluster nodes (manager node, peer nodes, and search heads) must run on the same operating system and version.

If the indexer cluster is integrated with a search head cluster, then the search head cluster instances, including the deployer, must run on the same operating system and version as the indexer cluster nodes.

---

Or is that not a hard and fast rule?

If it's the case that I can mix OS versions in an indexer cluster, in light of that, do you think it would be best to join the new indexers into the same cluster as the old indexers and let data replicate between them before retiring the old indexers? Or do you think it would be best to go with my original plan of dividing them into two separate clusters, send new data to the new cluster and search across both clusters until the old data ages off?

Thanks for the tip about the SH version, if I go with the second option I will need to upgrade that first then.

[PickleRick]

I would interpret it as "the same operating system and (splunk) version". So you must have - for example - splunk 8.1.3 across all your nodes and they all have to be either windows-based or linux-based.