Upgrading Distributed Environments Without Data Loss

[act_waltersj]

I'm taking over Splunk admin duties from a co-worker that has left the company.  We have a distributed environment setup of two heavy forwarders and four index servers all running  Splunk Enterprise 7.3.  I'm fleshing out our upgrade task list for moving to 8.1.2.  What is the best way to manage the upgrade across all servers, and do I need to take any special steps to prevent loss of logs?  I have the task for the install itself, I need a game plan for how to do it across the 6 servers.  Will upgrading each server - one at a time, have it back up and running before I move to the next server - prevent log/data loss, or is it better practice to do all 6 servers at once (take them all down, upgrade them all, the bring them all back online).  Looking at the compatibility matrix the I should be able to do all the indexers first and have the heavy forwarders still be compatible.  Any advice on how to manage this is greatly appreciated.

[act_waltersj]

Thanks for the information, Giuseppe.  We do not have clustered indexers but we do have one primary search head.  I've checked compatibility on the forwarders, so it appears one at a time, indexers then forwards is the way to go.  Thank you!

[gcusello · Accepted solution]

Hi @act_waltersj,

only two questions:

• have you clustered or not clustered Indexers?
• have you Search Heads?

Anyway, the upgrade order is the following:

• Master Node (if clustered Indexers);
• Search Heads;
• Indexers;
• Heavy Forwarders:
• Universal Forwarders.

If you have an Indexer Cluster, you can perform a rolling upgrade so you don't loose any data.

If you don't have an Indexer Cluster, you have to upgrade indexers only one a time.

You must put the same attention to Heavy Forwarders: upgrade only one a time.

Check the version of your Universal Forwarders to verify the compatibility with Indexers 8.1.2.

Ciao.

Giuseppe