In process of upgrading to 8.2.2.1 from 7.x.x any great lessons to follow? Should we upgrade the ES at the same time?

[SamHTexas]

Work in a large environment including Splunk Ent. & ES. Planning to upgrade from 7.x.x to 8.2.2.1. Any optimizations to perform ? Any best practices to follow? Should we upgrade the ES (Enterprise Security 6.4) before or after the Splunk Enterprise upgrade. Thanks a million for your help in advance.

[ro_mc]

To start with, this document is your new best friend:

https://docs.splunk.com/Documentation/VersionCompatibility/current/Matrix/CompatMatrix

Out of Splunk Enterprise and Enterprise Security, you'll want to upgrade Splunk Enterprise first, with a caveat in the section below...

If you're currently running Splunk ES 6.4 with Splunk 7.x.x, you'll notice that this isn't considered compatible, and you may run into problems during the upgrade. Splunk recommends updating Splunk Enterprise and ES in the same change window, but in your case, you should focus on getting Splunk Enterprise updated to a compatible state and supported edition before progressing further.

Normally, it would also matter whether you're running ES 6.4.0 or 6.4.1, as Splunk 8.2.2 is compatible with 6.4.1 but not 6.4.0. However, again, the lack of compatibility could cause problems so you should upgrade to 8.1 first, and then 8.2 per the documentation below:

https://docs.splunk.com/Documentation/Splunk/8.2.2/Installation/HowtoupgradeSplunk

Next, read the instructions, and here comes the caveat. The biggest change to Splunk 8 is the use of Python 3, and there is an app that lets you verify app compatibility in the new Splunk version. If your apps don't support Python 3 they may stop working. This is particularly important if Splunk is integrated with older third party applications, as some apps contain APIs that are application-version-specific. It's also relevant for any custom apps that have been written, or for apps that are no longer supported or have (still) not been upgraded for Python 3 compatibility (yes, there are a few).

In short, read the manual and know your environment before proceeding.

https://docs.splunk.com/Documentation/Splunk/8.0.2/Installation/AboutupgradingREADTHISFIRST

In terms of what infrastructure components to upgrade first, you should take a look at this resource, and it should form the basis for your upgrade plan:

https://community.splunk.com/t5/Installation/What-s-the-order-of-operations-for-upgrading-Splunk-Ent...

Basically, upgrade management components first, then search heads, then indexers. Note the verification between each management component upgrade. Ensure you have snapshots available and backups of data where appropriate, including backups of the KV store on the search head, as this is crucial to Enterprise Security.

Once you're up to Splunk 8.1.x on all servers, it's fairly smooth sailing. The upgrade to Splunk 8.2.x is fairly minor in comparison with a few great benefits, and then you can upgrade ES over the top of the existing version. There are optimisations available (1) for the KV store, which is highly recommended (the old engine will become obsolete), and (2) for improving indexing with the tsidx writing level, which you can read about once you get to that stage.

It's probably the biggest Splunk / ES upgrade you'll need to do for quite a while so triple check your documentation, get plenty of support from your leadership and management, and give yourself plenty of time to perform the change and get it done right.