Is it better to put peer offline or stop them when migrating a Splunk Enterprise instance?

[giulioBalza]

Hello,

before the upgradation to splunk 9.x we have to move the actual instances of Splunk to new VMs with new OS version and additional resources  (CPU, RAM and disk space [indexers]):

cluster SH: 3 nodes

cluster indexers: 16 peer (2 sites).

Migration for nodes like SHs, CM and Deploy are pretty clear, I have some doubt about the peers node.

Probably we do the migration a peer at time, is it better to put peer offline or stop them ?

in case is preferable to put them offline, is it possible to extend the restart period without time limit, for example 9 hours or more ? This is due to syncing the file system where indexes sits to the new VM.

It also not clear if offline method rolls the bucket to warm from hot or must be done manually

Thanks,

[richgalloway]

There is Splunk documentation on migrating to new hardware at https://docs.splunk.com/Documentation/Splunk/9.0.2/Installation/MigrateaSplunkinstance .  Those docs are incomplete, however, especially with regard to upgrading the hardware on an indexer cluster.

For indexer clusters, the best method is to add new indexers to the existing cluster on new hardware/VMs.  Put the old indexers into manual detention and shut them down one at a time using the command

splunk offline --enforce-counts

Wait for the indexer to stop itself before shutting down the next one.  As each indexer shuts down, the Cluster Manager will ensure each primary bucket is replicated on one of the new indexers.

[gballanti]

Hello Rick,

thank you for the reply.

Unfortunately even if your solution is really smart and having the same issue as GiulioBalza probably i can't do that, because if I'm not wrong a rebalance is needed after i added a new peer to the cluster.

Currently due to a bug related to the excessive bucket removal the Cluster Master has a newer version than peers that stops data rebalancing.

Regards,

G

[richgalloway · Accepted solution]

Rebalance is not needed after adding the new peers in this case because buckets will be copied to the new indexers when the old indexers shut down.  It's a good idea, however, to rebalance after all of the old indexers are removed.