Installation

Any recommendations on what order to upgrade my Splunk Instances from 6.4.0 to 6.4.4?

anaqvi
Explorer

Any recommendations on what order and how to upgrade the instances from 6.4.0 to 6.4.4. I currently have the following instances:
1 Deployment Server
1 Cluster Master (Also the license Master)
4 Indexers (Clustered)
1 Deployer
3 SH (Pooled + Distributed Search)
1 SH (non-pooled)
1 Staging testserver
1 ES staging instance

Labels (3)
0 Karma
1 Solution

ekost
Splunk Employee
Splunk Employee

Splunk is generally upgraded from the SH tier down. The ES staging instance implies you’re running ES on the SHC.

  1. Verify the version of ES you’re running supports installation on 6.4.4. If not, upgrade it to a version that does.
  2. Upgrade the CM first. I added this step to clarify the order-of-ops process defined in the docs.
  3. Upgrade all SHC nodes, and upgrade the deployer following the documented steps.
  4. Place the CM into maintenance mode and upgrade the index cluster. All clustered indexers should be taken down for the upgrade, as at this time upgrading some indexers while leaving others running is only supported for maintenance releases (e.g. 6.4.1 to 6.4.1.1.)
  5. Disable maintenance mode on the CM. Note: you must finish the upgrade on all indexer nodes before disabling maintenance mode.
  6. Upgrade supporting nodes such as the deployment server.
  7. Upgrade any staging instances.

I have linux hosts so I’ll use a distributed shell to apply the same command across multiple nodes:
1. Grab a backup of the core Splunk configurations. You can copy the folder on each node to a new folder (ugh! And it assumes that any index buckets are NOT under %splunk_home%,) or just run diags to keep a copy.
2. Follow the upgrade instruction noted above per-tier: Untar the latest release on top of the old installation. Start services, and check for errors. Bring the tiers back up in the order suggested in the docs. NOTE: Due to special restrictions on clustered nodes, read the doc links above carefully as there are order-of-ops nuances. Your clusters will be down for a time, so make sure you understand what data sources in your environment will show gaps. For example, data polled for indexing using a script may miss an interval, any UDP data sources going to a Splunk instance you stop services on will not be indexed, etc.
3. Check that the basic services are working: LDAP logins, key apps, scripted inputs, critical data source checks, forwarder management has check-ins, license server is shows all’s well, CM show’s all’s well, etc.
4. This is a brilliant time to document the instance-specific details of your upgrade process. Include the validation checks you ran in your docs.

View solution in original post

jmulcaster_splu
Splunk Employee
Splunk Employee

FYI, we've posted an upgrade roadmap with links to the latest documentation to help with upgrade planning. Check it out and let us know if you find it helpful. What's the order of operations for upgrading Splunk Enterprise?

0 Karma

ekost
Splunk Employee
Splunk Employee

Splunk is generally upgraded from the SH tier down. The ES staging instance implies you’re running ES on the SHC.

  1. Verify the version of ES you’re running supports installation on 6.4.4. If not, upgrade it to a version that does.
  2. Upgrade the CM first. I added this step to clarify the order-of-ops process defined in the docs.
  3. Upgrade all SHC nodes, and upgrade the deployer following the documented steps.
  4. Place the CM into maintenance mode and upgrade the index cluster. All clustered indexers should be taken down for the upgrade, as at this time upgrading some indexers while leaving others running is only supported for maintenance releases (e.g. 6.4.1 to 6.4.1.1.)
  5. Disable maintenance mode on the CM. Note: you must finish the upgrade on all indexer nodes before disabling maintenance mode.
  6. Upgrade supporting nodes such as the deployment server.
  7. Upgrade any staging instances.

I have linux hosts so I’ll use a distributed shell to apply the same command across multiple nodes:
1. Grab a backup of the core Splunk configurations. You can copy the folder on each node to a new folder (ugh! And it assumes that any index buckets are NOT under %splunk_home%,) or just run diags to keep a copy.
2. Follow the upgrade instruction noted above per-tier: Untar the latest release on top of the old installation. Start services, and check for errors. Bring the tiers back up in the order suggested in the docs. NOTE: Due to special restrictions on clustered nodes, read the doc links above carefully as there are order-of-ops nuances. Your clusters will be down for a time, so make sure you understand what data sources in your environment will show gaps. For example, data polled for indexing using a script may miss an interval, any UDP data sources going to a Splunk instance you stop services on will not be indexed, etc.
3. Check that the basic services are working: LDAP logins, key apps, scripted inputs, critical data source checks, forwarder management has check-ins, license server is shows all’s well, CM show’s all’s well, etc.
4. This is a brilliant time to document the instance-specific details of your upgrade process. Include the validation checks you ran in your docs.

ekost
Splunk Employee
Splunk Employee

The latest documentation update for customers with both a SHC and an index cluster are here for 6.5.

0 Karma

ekost
Splunk Employee
Splunk Employee

The 6.4.4 documentation update for SHC and an index cluster is here.

0 Karma

gjanders
SplunkTrust
SplunkTrust

An excellent answer, and I have used a similar method in the past, however the documentation states a different order:
http://docs.splunk.com/Documentation/Splunk/6.5.0/Indexer/Upgradeacluster

I've already provided some feedback on that documentation but would you mind getting it updated if your above method is the preferred method of upgrade?

Note that my feedback was that the search head cluster upgrade instructions for newer 6.x versions advise to transfer captaincy one node at a time rather than stop all search heads.

However been clear on whether the cluster master/indexers must upgrade before the search heads would be great, and preferably this should be in the documentation !

0 Karma

ekost
Splunk Employee
Splunk Employee

I'll reconfirm the order-of-ops and run it by the docs team.

0 Karma

gjanders
SplunkTrust
SplunkTrust
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...