Getting Data In

Is it better to put peer offline or stop them when migrating a Splunk Enterprise instance?

giulioBalza
Path Finder

Hello,

before the upgradation to splunk 9.x we have to move the actual instances of Splunk to new VMs with new OS version and additional resources  (CPU, RAM and disk space [indexers]):

cluster SH: 3 nodes

cluster indexers: 16 peer (2 sites).

Migration for nodes like SHs, CM and Deploy are pretty clear, I have some doubt about the peers node.

Probably we do the migration a peer at time, is it better to put peer offline or stop them ?

in case is preferable to put them offline, is it possible to extend the restart period without time limit, for example 9 hours or more ? This is due to syncing the file system where indexes sits to the new VM.

It also not clear if offline method rolls the bucket to warm from hot or must be done manually

Thanks,

Labels (1)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

Rebalance is not needed after adding the new peers in this case because buckets will be copied to the new indexers when the old indexers shut down.  It's a good idea, however, to rebalance after all of the old indexers are removed.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

0 Karma

richgalloway
SplunkTrust
SplunkTrust

There is Splunk documentation on migrating to new hardware at https://docs.splunk.com/Documentation/Splunk/9.0.2/Installation/MigrateaSplunkinstance .  Those docs are incomplete, however, especially with regard to upgrading the hardware on an indexer cluster.

For indexer clusters, the best method is to add new indexers to the existing cluster on new hardware/VMs.  Put the old indexers into manual detention and shut them down one at a time using the command

splunk offline --enforce-counts

Wait for the indexer to stop itself before shutting down the next one.  As each indexer shuts down, the Cluster Manager will ensure each primary bucket is replicated on one of the new indexers.

---
If this reply helps you, Karma would be appreciated.
0 Karma

gballanti
Explorer

Hello Rick,

thank you for the reply.

Unfortunately even if your solution is really smart and having the same issue as GiulioBalza probably i can't do that, because if I'm not wrong a rebalance is needed after i added a new peer to the cluster.

Currently due to a bug related to the excessive bucket removal the Cluster Master has a newer version than peers that stops data rebalancing.

Regards,

G

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Rebalance is not needed after adding the new peers in this case because buckets will be copied to the new indexers when the old indexers shut down.  It's a good idea, however, to rebalance after all of the old indexers are removed.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...