Can we use the two sites of a multisite indexer cluster to improve the Splunk upgrade?

[ddrillic]

I wonder whether during a Splunk upgrade we can use our two sites of the multisite indexer cluster to improve the Splunk upgrade. Meaning, is it possible to upgrade the indexers of one site and then upgrade the indexers of the second site?

In our upgrade last week we simply upgraded all the indexers together and it took the indexers over 12 hours to catch up with the indexing load, replication and buckets fix-up.

It relates to Large amount of buckets that need to be fixed after the upgrade?

[FrankVl · Accepted solution]

Yes, you could have done a site-by-site upgrade: https://docs.splunk.com/Documentation/Splunk/7.3.0/Indexer/Upgradeacluster#Site-by-site_upgrade_for_...

And if you're already on a recent enough version (7.1.x or newer) you can even do a rolling upgrade of your indexers: https://docs.splunk.com/Documentation/Splunk/7.3.0/Indexer/Searchablerollingupgrade

[ddrillic]

@FrankVl, the documentation says -

-- When you initiate a rolling upgrade, you select a peer and take it offline. During the offline process, the master reassigns bucket primaries to other peers to retain the searchable state, and the peer completes any in-progress searches within a configurable timeout. See The fast offline process.

After the master shuts down the peer, you perform the software upgrade and bring the peer back online, at which point the peer rejoins the cluster. You repeat this process for each peer node until the rolling upgrade is complete.

Pretty amazing.

@FrankVl, we are not in maintenance mode, right?

[FrankVl]

Yes you are in maintenance mode during the rolling upgrade. See step 3: https://docs.splunk.com/Documentation/Splunk/latest/Indexer/Searchablerollingupgrade#3._Initialize_r...

3. Initialize rolling upgrade
Run the following CLI command on the cluster master:
    splunk upgrade-init cluster-peers 
Or, send a POST request to the following endpoint:
    cluster/master/control/control/rolling_upgrade_init
This initializes the rolling upgrade and puts the cluster in maintenance mode.

[ddrillic]

Makes sense - thank you @FrankVl.

[ddrillic]

Gorgeous @FrankVl.

[FrankVl]

Please mark the answer as accepted if it solves your question.

[ddrillic]

From the SE -

-- Looks like you have your answers already. Site-by-site upgrades work in most cases.

There are caveats with certain breaking change type upgrades, but those are really rare, the most recent one I can remember is the SSL version change.

7.1+ does also allow for rolling upgrades as mentioned in the answers responses. This is likely your best path forward.