A customer had been trying to upgrade our Test Splunk multisite environment from 6.3 to 6.5.1 but was unable to progress any further once the customer upgraded one site and attempted to let data replicate across as mentioned in the documentation located at 'http://docs.splunk.com/Documentation/Splunk/6.5.2/Indexer/Upgradeacluster#Site-by-site_upgrade_for_multisite_indexer_clusters', which are noted below:
1 Upgrade of the master node.
Upgrade of the site1 peers and search heads.
Upgrade of the site2 peers and search heads.
Here are the steps in detail:
Stop the master.
Upgrade the master node, following the normal procedure for any Splunk Enterprise upgrade, as described in How to upgrade Splunk Enterprise in the Installation Manual.
Start the master, accepting all prompts, if it is not already running.
Run splunk enable maintenance-mode on the master. To confirm that the master is in maintenance mode, run splunk show maintenance-mode. This step prevents unnecessary bucket fix-ups. See Use maintenance mode.
Stop all the peers and search heads on site1 with the splunk stop command.
Upgrade the site1 peer nodes and search heads.
Start the site1 peer nodes and search heads, if they are not already running.
Run splunk disable maintenance-mode on the master. To confirm that the master is not in maintenance mode, run splunk show maintenance-mode.
Wait until the master dashboard shows that both the search factor and replication factor are met.
Run splunk enable maintenance-mode on the master. To confirm that the master is in maintenance mode, run splunk show maintenance-mode.
Stop all the peers and search heads on site2 with the splunk stop command.
Upgrade the site2 peer nodes and search heads.
Start the site2 peer nodes and search heads, if they are not already running.
Run splunk disable maintenance-mode on the master. To confirm that the master is not in maintenance mode, run splunk show maintenance-mode.
Once customer got to step 9 'Wait until the master dashboard shows that both the search factor and replication factor are met.' we noticed that the search and replication factor were never being met. Below can be considered an alternative, or perhaps 'better' way to achieve a multi-site indexer clustering upgrade.
... View more