Which of these methods of adding new upgraded indexers to my cluster would be best?

[Bomo2023]

I currently have a Splunk cluster that looks like this:

Splunk	CentOS Version	Splunk Version
Master	7.5	7.0.0
Forwarder	7.5	Universal Forwarder 6.6.3
Search Head	6.5	7.0.0
Indexer 1	6.5	7.0.0
Indexer 2	6.5	7.0.0
Indexer 3	6.5	7.0.0
Indexer 4	6.5	7.0.0

I have 4 new, better servers that I want to build as indexers to replace the 4 indexers that I currently have (while moving from Splunk 7.0.0 to Splunk 8.x in the process).

It seems to me that I have two options:

Option 1
Create a new indexer cluster with the 4 new indexers, so that I have an old indexer cluster and a new indexer cluster.  Send new data to the new indexer cluster only and in time the data on the old indexer cluster will age off and I can retire that cluster. During this time I will configure the search head to search across both indexer clusters so that the old and new data is searchable.

Option 2
Add the new indexers to the existing indexer cluster, so that I have 4 old indexers and 4 new indexers in the same cluster. Allow the data to replicate across the 8 node cluster, and then retire the old indexers one by one. Leaving me with the 4 new indexers only.

In both scenarios above the old indexers will be CentOS 6.x and the new indexers will be CentOS 7.x, but as I understand it this won't be a problem.

Would anyone be able to offer any advice on which method would be best/easiest or point out any pitfalls or anything obvious that I may not have considered?

[richgalloway · Accepted solution]

Use Option 2.  Once the new indexers are added to the cluster, put the old indexers into detention.  Then run splunk offline --enforce-counts on one old indexer at a time.  The CM will make sure data from the old indexer is moved to the new indexers.  Wait until the old indexer stops before running the command on the next old indexer.

[isoutamo]

I also prefer the option 2. It will be much quicker way to get rid of old boxes. Also remember to rebalance buckets after you have removed old nodes from cluster.

Here https://community.splunk.com/t5/Splunk-Enterprise/Migration-of-Splunk-to-different-server-same-platf... is how I have done it as part of migrating multi site environment from one service provider to another.

r. Ismo