Installation

Any recommendations on what order to upgrade my Splunk Instances from 6.4.0 to 6.4.4?

anaqvi
Explorer

Any recommendations on what order and how to upgrade the instances from 6.4.0 to 6.4.4. I currently have the following instances:
1 Deployment Server
1 Cluster Master (Also the license Master)
4 Indexers (Clustered)
1 Deployer
3 SH (Pooled + Distributed Search)
1 SH (non-pooled)
1 Staging testserver
1 ES staging instance

Labels (3)
0 Karma
1 Solution

ekost
Splunk Employee
Splunk Employee

Splunk is generally upgraded from the SH tier down. The ES staging instance implies you’re running ES on the SHC.

  1. Verify the version of ES you’re running supports installation on 6.4.4. If not, upgrade it to a version that does.
  2. Upgrade the CM first. I added this step to clarify the order-of-ops process defined in the docs.
  3. Upgrade all SHC nodes, and upgrade the deployer following the documented steps.
  4. Place the CM into maintenance mode and upgrade the index cluster. All clustered indexers should be taken down for the upgrade, as at this time upgrading some indexers while leaving others running is only supported for maintenance releases (e.g. 6.4.1 to 6.4.1.1.)
  5. Disable maintenance mode on the CM. Note: you must finish the upgrade on all indexer nodes before disabling maintenance mode.
  6. Upgrade supporting nodes such as the deployment server.
  7. Upgrade any staging instances.

I have linux hosts so I’ll use a distributed shell to apply the same command across multiple nodes:
1. Grab a backup of the core Splunk configurations. You can copy the folder on each node to a new folder (ugh! And it assumes that any index buckets are NOT under %splunk_home%,) or just run diags to keep a copy.
2. Follow the upgrade instruction noted above per-tier: Untar the latest release on top of the old installation. Start services, and check for errors. Bring the tiers back up in the order suggested in the docs. NOTE: Due to special restrictions on clustered nodes, read the doc links above carefully as there are order-of-ops nuances. Your clusters will be down for a time, so make sure you understand what data sources in your environment will show gaps. For example, data polled for indexing using a script may miss an interval, any UDP data sources going to a Splunk instance you stop services on will not be indexed, etc.
3. Check that the basic services are working: LDAP logins, key apps, scripted inputs, critical data source checks, forwarder management has check-ins, license server is shows all’s well, CM show’s all’s well, etc.
4. This is a brilliant time to document the instance-specific details of your upgrade process. Include the validation checks you ran in your docs.

View solution in original post

jmulcaster_splu
Splunk Employee
Splunk Employee

FYI, we've posted an upgrade roadmap with links to the latest documentation to help with upgrade planning. Check it out and let us know if you find it helpful. What's the order of operations for upgrading Splunk Enterprise?

0 Karma

ekost
Splunk Employee
Splunk Employee

Splunk is generally upgraded from the SH tier down. The ES staging instance implies you’re running ES on the SHC.

  1. Verify the version of ES you’re running supports installation on 6.4.4. If not, upgrade it to a version that does.
  2. Upgrade the CM first. I added this step to clarify the order-of-ops process defined in the docs.
  3. Upgrade all SHC nodes, and upgrade the deployer following the documented steps.
  4. Place the CM into maintenance mode and upgrade the index cluster. All clustered indexers should be taken down for the upgrade, as at this time upgrading some indexers while leaving others running is only supported for maintenance releases (e.g. 6.4.1 to 6.4.1.1.)
  5. Disable maintenance mode on the CM. Note: you must finish the upgrade on all indexer nodes before disabling maintenance mode.
  6. Upgrade supporting nodes such as the deployment server.
  7. Upgrade any staging instances.

I have linux hosts so I’ll use a distributed shell to apply the same command across multiple nodes:
1. Grab a backup of the core Splunk configurations. You can copy the folder on each node to a new folder (ugh! And it assumes that any index buckets are NOT under %splunk_home%,) or just run diags to keep a copy.
2. Follow the upgrade instruction noted above per-tier: Untar the latest release on top of the old installation. Start services, and check for errors. Bring the tiers back up in the order suggested in the docs. NOTE: Due to special restrictions on clustered nodes, read the doc links above carefully as there are order-of-ops nuances. Your clusters will be down for a time, so make sure you understand what data sources in your environment will show gaps. For example, data polled for indexing using a script may miss an interval, any UDP data sources going to a Splunk instance you stop services on will not be indexed, etc.
3. Check that the basic services are working: LDAP logins, key apps, scripted inputs, critical data source checks, forwarder management has check-ins, license server is shows all’s well, CM show’s all’s well, etc.
4. This is a brilliant time to document the instance-specific details of your upgrade process. Include the validation checks you ran in your docs.

ekost
Splunk Employee
Splunk Employee

The latest documentation update for customers with both a SHC and an index cluster are here for 6.5.

0 Karma

ekost
Splunk Employee
Splunk Employee

The 6.4.4 documentation update for SHC and an index cluster is here.

0 Karma

gjanders
SplunkTrust
SplunkTrust

An excellent answer, and I have used a similar method in the past, however the documentation states a different order:
http://docs.splunk.com/Documentation/Splunk/6.5.0/Indexer/Upgradeacluster

I've already provided some feedback on that documentation but would you mind getting it updated if your above method is the preferred method of upgrade?

Note that my feedback was that the search head cluster upgrade instructions for newer 6.x versions advise to transfer captaincy one node at a time rather than stop all search heads.

However been clear on whether the cluster master/indexers must upgrade before the search heads would be great, and preferably this should be in the documentation !

0 Karma

ekost
Splunk Employee
Splunk Employee

I'll reconfirm the order-of-ops and run it by the docs team.

0 Karma

gjanders
SplunkTrust
SplunkTrust
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...