Upgrade Splunk 6.6.3 to 7.2.3

[jeffbat]

I am trying to figure out if I will run into any issues while upgrading our Splunk Enterprise environment from 6.6.3 to 7.2.3.

We have a distributed environment that has:
• License Master (covers all environments)
• Search Head (test)
• Indexer (test)
• 2 Heavy Forwarders (test)
• Search Head Cluster (4 nodes with an additional Deployer server – which is also the Cluster Master for the West Coast datacenter) (prod)
• Stand Alone Search Head (prod)
• 2 Heavy Forwarders (prod)
• Index Cluster (4 nodes with additional Cluster Master server) (prod) (West Coast datacenter)
• Index Cluster (4 nodes with additional Cluster Master server) (prod) (East Coast datacenter)
• Index Cluster (3 nodes with additional Cluster Master server)
o The Cluster Master is also the Deployment Server for both prod and test environment
• Search Head running Enterprise Security

We currently have a few caveats in the environment that will affect our upgrade. We cannot upgrade Enterprise Security for now which means that we cannot upgrade the Search Head it runs on since our ES version is 4.7.4 which cannot run with Splunk 7.2.3.

My plan is to upgrade in the following order:
• License Master
• Test Search Head
• Test Indexer
• Test Heavy Forwarders (both)
• Prod Stand Alone Search Head
• All 3 Cluster Masters
o 1 is also the Deployer for the Search Head cluster
o 1 is also the Deployment Server
• Prod Search head Cluster

This will leave the Prod Heavy Forwarders and all of the Prod Indexers on Splunk 6.6.3. We will also not upgrade any of our Universal Forwarders until are able to move forward with updating the rest of the infrastructure servers.

Does this plan look to cover everything or we have problems with it?

Thanks.

[nickhills]

If you upgrade your indexers before the attached search heads, you will be running an incompatible setup (and you will get constant warnings on all searches from the affected SH).

Your upgrades should always start with search heads.
If this is a blocker (because ES etc) then you should consider deferring the upgrade until you are in a position to update ES to a version which supports your target Splunk Core version.

You should also not run a CM on a later version than it's peers.
SHC and IDX clusters have to be updated at once, and that includes SHC Deployers, and IDX Cluster Masters, as well as the peers.

I feel your pain. I have been there!

[jeffbat]

From what I read on this link

https://docs.splunk.com/Documentation/Splunk/7.2.4/Indexer/Upgradeacluster#Upgrade_each_tier_separat...

It says that you can update each tier of the clustered index/search separately by going master, search heads, then indexers. I do see that it is not recommended to stay in a config of different versions for lengths of time but does seem to allow for that.

Am I just reading that incorrectly?

[nickhills]

I think you have read that correctly, but I have had (albeit an 6.x release) cluster peers go inconsistent (unrelated to the upgrade work) after I upgraded the master.

The fixup never completed, and we eventually theorised that it was because the master was at a later rev. An emergency peer upgrade later, and consistency returned.

It maybe that 7.x handles this better, but there is the following note at the top of that page:

Caution: Even when upgrading each tier separately, it is strongly recommended that you complete the entire upgrade process quickly, to avoid any possibility of incompatibilities between node types running different versions.

[nickhills]

.... sorry, submitted before I had finished....

I guess it all depends how quickly you can move.. if you plan to complete the whole process in hours or days, your approach is probably fine, but I’d be reluctant if your schedule is measured in weeks.

[jeffbat]

It would be weeks as our security team is moving away from Enterprise Security and I just didn't want to have to upgrade ES on its search head and risk something not going cleanly and then have to spend so much cycles getting it back and working before we remove ES.

We are discussing of just going ahead and doing the upgrade of ES on that server so that we can get the rest of the environment can be upgraded.