Installation

Any recommendations on what order to upgrade my Splunk Instances from 6.4.0 to 6.4.4?

anaqvi
Explorer

Any recommendations on what order and how to upgrade the instances from 6.4.0 to 6.4.4. I currently have the following instances:
1 Deployment Server
1 Cluster Master (Also the license Master)
4 Indexers (Clustered)
1 Deployer
3 SH (Pooled + Distributed Search)
1 SH (non-pooled)
1 Staging testserver
1 ES staging instance

Labels (3)
0 Karma
1 Solution

ekost
Splunk Employee
Splunk Employee

Splunk is generally upgraded from the SH tier down. The ES staging instance implies you’re running ES on the SHC.

  1. Verify the version of ES you’re running supports installation on 6.4.4. If not, upgrade it to a version that does.
  2. Upgrade the CM first. I added this step to clarify the order-of-ops process defined in the docs.
  3. Upgrade all SHC nodes, and upgrade the deployer following the documented steps.
  4. Place the CM into maintenance mode and upgrade the index cluster. All clustered indexers should be taken down for the upgrade, as at this time upgrading some indexers while leaving others running is only supported for maintenance releases (e.g. 6.4.1 to 6.4.1.1.)
  5. Disable maintenance mode on the CM. Note: you must finish the upgrade on all indexer nodes before disabling maintenance mode.
  6. Upgrade supporting nodes such as the deployment server.
  7. Upgrade any staging instances.

I have linux hosts so I’ll use a distributed shell to apply the same command across multiple nodes:
1. Grab a backup of the core Splunk configurations. You can copy the folder on each node to a new folder (ugh! And it assumes that any index buckets are NOT under %splunk_home%,) or just run diags to keep a copy.
2. Follow the upgrade instruction noted above per-tier: Untar the latest release on top of the old installation. Start services, and check for errors. Bring the tiers back up in the order suggested in the docs. NOTE: Due to special restrictions on clustered nodes, read the doc links above carefully as there are order-of-ops nuances. Your clusters will be down for a time, so make sure you understand what data sources in your environment will show gaps. For example, data polled for indexing using a script may miss an interval, any UDP data sources going to a Splunk instance you stop services on will not be indexed, etc.
3. Check that the basic services are working: LDAP logins, key apps, scripted inputs, critical data source checks, forwarder management has check-ins, license server is shows all’s well, CM show’s all’s well, etc.
4. This is a brilliant time to document the instance-specific details of your upgrade process. Include the validation checks you ran in your docs.

View solution in original post

jmulcaster_splu
Splunk Employee
Splunk Employee

FYI, we've posted an upgrade roadmap with links to the latest documentation to help with upgrade planning. Check it out and let us know if you find it helpful. What's the order of operations for upgrading Splunk Enterprise?

0 Karma

ekost
Splunk Employee
Splunk Employee

Splunk is generally upgraded from the SH tier down. The ES staging instance implies you’re running ES on the SHC.

  1. Verify the version of ES you’re running supports installation on 6.4.4. If not, upgrade it to a version that does.
  2. Upgrade the CM first. I added this step to clarify the order-of-ops process defined in the docs.
  3. Upgrade all SHC nodes, and upgrade the deployer following the documented steps.
  4. Place the CM into maintenance mode and upgrade the index cluster. All clustered indexers should be taken down for the upgrade, as at this time upgrading some indexers while leaving others running is only supported for maintenance releases (e.g. 6.4.1 to 6.4.1.1.)
  5. Disable maintenance mode on the CM. Note: you must finish the upgrade on all indexer nodes before disabling maintenance mode.
  6. Upgrade supporting nodes such as the deployment server.
  7. Upgrade any staging instances.

I have linux hosts so I’ll use a distributed shell to apply the same command across multiple nodes:
1. Grab a backup of the core Splunk configurations. You can copy the folder on each node to a new folder (ugh! And it assumes that any index buckets are NOT under %splunk_home%,) or just run diags to keep a copy.
2. Follow the upgrade instruction noted above per-tier: Untar the latest release on top of the old installation. Start services, and check for errors. Bring the tiers back up in the order suggested in the docs. NOTE: Due to special restrictions on clustered nodes, read the doc links above carefully as there are order-of-ops nuances. Your clusters will be down for a time, so make sure you understand what data sources in your environment will show gaps. For example, data polled for indexing using a script may miss an interval, any UDP data sources going to a Splunk instance you stop services on will not be indexed, etc.
3. Check that the basic services are working: LDAP logins, key apps, scripted inputs, critical data source checks, forwarder management has check-ins, license server is shows all’s well, CM show’s all’s well, etc.
4. This is a brilliant time to document the instance-specific details of your upgrade process. Include the validation checks you ran in your docs.

ekost
Splunk Employee
Splunk Employee

The latest documentation update for customers with both a SHC and an index cluster are here for 6.5.

0 Karma

ekost
Splunk Employee
Splunk Employee

The 6.4.4 documentation update for SHC and an index cluster is here.

0 Karma

gjanders
SplunkTrust
SplunkTrust

An excellent answer, and I have used a similar method in the past, however the documentation states a different order:
http://docs.splunk.com/Documentation/Splunk/6.5.0/Indexer/Upgradeacluster

I've already provided some feedback on that documentation but would you mind getting it updated if your above method is the preferred method of upgrade?

Note that my feedback was that the search head cluster upgrade instructions for newer 6.x versions advise to transfer captaincy one node at a time rather than stop all search heads.

However been clear on whether the cluster master/indexers must upgrade before the search heads would be great, and preferably this should be in the documentation !

0 Karma

ekost
Splunk Employee
Splunk Employee

I'll reconfirm the order-of-ops and run it by the docs team.

0 Karma

gjanders
SplunkTrust
SplunkTrust
0 Karma

jagadeeshm
Contributor
0 Karma
Get Updates on the Splunk Community!

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...