How to properly upgrade a multi-site indexer cluster

[kgrigsby_splunk]

A customer had been trying to upgrade our Test Splunk multisite environment from 6.3 to 6.5.1 but was unable to progress any further once the customer upgraded one site and attempted to let data replicate across as mentioned in the documentation located at 'http://docs.splunk.com/Documentation/Splunk/6.5.2/Indexer/Upgradeacluster#Site-by-site_upgrade_for_m...', which are noted below:

1 Upgrade of the master node.

1. Upgrade of the site1 peers and search heads.

2. Upgrade of the site2 peers and search heads.

Here are the steps in detail:

1. Stop the master.

2. Upgrade the master node, following the normal procedure for any Splunk Enterprise upgrade, as described in How to upgrade Splunk Enterprise in the Installation Manual.

3. Start the master, accepting all prompts, if it is not already running.

4. Run splunk enable maintenance-mode on the master. To confirm that the master is in maintenance mode, run splunk show maintenance-mode. This step prevents unnecessary bucket fix-ups. See Use maintenance mode.

5. Stop all the peers and search heads on site1 with the splunk stop command.

6. Upgrade the site1 peer nodes and search heads.

7. Start the site1 peer nodes and search heads, if they are not already running.

8. Run splunk disable maintenance-mode on the master. To confirm that the master is not in maintenance mode, run splunk show maintenance-mode.

9. Wait until the master dashboard shows that both the search factor and replication factor are met.

10. Run splunk enable maintenance-mode on the master. To confirm that the master is in maintenance mode, run splunk show maintenance-mode.

11. Stop all the peers and search heads on site2 with the splunk stop command.

12. Upgrade the site2 peer nodes and search heads.

13. Start the site2 peer nodes and search heads, if they are not already running.

14. Run splunk disable maintenance-mode on the master. To confirm that the master is not in maintenance mode, run splunk show maintenance-mode.

• Once customer got to step 9 'Wait until the master dashboard shows that both the search factor and replication factor are met.' we noticed that the search and replication factor were never being met. Below can be considered an alternative, or perhaps 'better' way to achieve a multi-site indexer clustering upgrade.

[Steve_G_]

We only support rolling multisite upgrades from one minor version to the next. That is, we support 6.3 to 6.4, and 6.4 to 6.5, but not 6.3 to 6.5.

With single version upgrades, the instructions in the documentation should work properly, and they have the advantage of allowing the cluster to return to a complete state more quickly than it would if you were to keep maintenance mode turned on for the entire duration of the upgrade process. There's no guarantee that leaving the cluster in maintenance mode for the entire upgrade process would always remediate issues arising from skip-version upgrades, in any case.

The docs did not previously note the restriction regarding single-version upgrades. They do now: See http://docs.splunk.com/Documentation/Splunk/6.5.2/Indexer/Upgradeacluster#Site-by-site_upgrade_for_m... -- "If you have a multisite cluster, you can upgrade one site at a time, as long as you are upgrading across no more than one minor version (for example, from 6.4 to 6.5)."

If you need to perform a skip-version upgrade, you have two choices:

• You can perform a non-rolling upgrade of the cluster, as described in http://docs.splunk.com/Documentation/Splunk/6.5.2/Indexer/Upgradeacluster#Upgrade_a_6.x_indexer_clus....
• You can perform a rolling upgrade in two steps. That is, to go from 6.3 to 6.5, you can upgrade all nodes from 6.3 to 6.4, followed by a second upgrade from 6.4 to 6.5.

[gjanders]

kgrigsby has provided a thorough answer, you may also want to refer to the official Upgrade an indexer cluster there is a specific subsection on "Site-by-site upgrade for multisite indexer clusters".

[kgrigsby_splunk]

The proper way to execute a multi-site indexer cluster upgrade is to keep the master in maintenance mode throughout the upgrade. The steps below more closely resemble this method with the following deployment:

1 – Master
1 – Deployer
4 Indexers – Multisite Cluster ( 2 each)
3 SH ( 1 on Site 1 + 2 on Site 2)
2 HFs

1. Stop the master.

2. Upgrade the master node, following the normal procedure for any Splunk Enterprise upgrade, as described in How to upgrade Splunk Enterprise in the Installation Manual 'http://docs.splunk.com/Documentation/Splunk/6.5.2/Installation/HowtoupgradeSplunk.'

3. Start the master, accepting all prompts, if it is not already running.

4. Run 'splunk enable maintenance-mode' on the master. To confirm that the master is in maintenance mode, run 'splunk show maintenance-mode'. This step prevents unnecessary bucket fix-ups. See Use maintenance mode 'http://docs.splunk.com/Documentation/Splunk/6.5.2/Indexer/Usemaintenancemode'.

5. Stop all the peers and search heads on site1 with the splunk stop command.

6. Upgrade the site1 peer nodes and search heads.

7. Start the site1 peer nodes and search heads, if they are not already running.

8. Stop search heads on site2 with the splunk stop command.

9. Upgrade the site2 search heads.

10. Start the site2 search heads.

11. Stop 1 site2 peer.

12. Upgrade that site2 peer

13. Start that site2 peer

14. Stop the second site2 peer

15. Upgrade the second site2 peer

16. Start the second site2 peer

17. Run 'splunk disable maintenance-mode' on the master. To confirm that the master is not in maintenance mode, run 'splunk show maintenance-mode'.

18. Wait until the master dashboard shows that both the search factor and replication factor are met.

19. If there are 'streaming' errors at this point, run 'splunk rolling-restart cluster-peers' in order to remove the errors.

[vguptadevops]

@kgrigsby_splunk Many thanks for this detailed answer , But I just want to ask q question regards to the people who are upgrading a major version like from 6.6 to 7.0.x build , we have a similar requirement and we are also running in multisite configuration where we just have 1 Cluster Master , 2 search heads [ 1 for each site] and 4 indexers [ 2 for each site with indexer cluster]. If I go by above process I can just upgrade master and search head and search peers for site 1 and then the same for site 2 . The only confusion I am running into is that would it be ok for me to keep the indexer cluster & Mater shutoff completely during the upgrade, what is the implication of me upgrading directly from 6.6 to 7.0.3 version.

Thanks

[richgalloway]

@vguptadevops This question is more than 2 years old. Please post a new question describing your problem.

[vguptadevops]

Hi @richgalloway Thanks I will do so , but I was just wondering if there is any such link where the major version upgrade is explained , I am not sure if there is any and above answer seems still valid in my environment.
Do you have any suggestion there?

[richgalloway]

Please post a new question instead of hijacking an existing thread.

[acharlieh]

@kgrigsby_splunk if you think there should be a change to the docs, posting a question and self-answering here is great, but what would be awesome too is if you submit feedback on the docs page where you think the change should be made under "Was this topic useful?" at the bottom of the page. @ChrisG and the docs team are amazingly responsive to feedback.