Any recommendations on what order and how to upgrade the instances from 6.4.0 to 6.4.4. I currently have the following instances:
1 Deployment Server
1 Cluster Master (Also the license Master)
4 Indexers (Clustered)
1 Deployer
3 SH (Pooled + Distributed Search)
1 SH (non-pooled)
1 Staging testserver
1 ES staging instance
Splunk is generally upgraded from the SH tier down. The ES staging instance implies you’re running ES on the SHC.
I have linux hosts so I’ll use a distributed shell to apply the same command across multiple nodes:
1. Grab a backup of the core Splunk configurations. You can copy the folder on each node to a new folder (ugh! And it assumes that any index buckets are NOT under %splunk_home%,) or just run diags to keep a copy.
2. Follow the upgrade instruction noted above per-tier: Untar the latest release on top of the old installation. Start services, and check for errors. Bring the tiers back up in the order suggested in the docs. NOTE: Due to special restrictions on clustered nodes, read the doc links above carefully as there are order-of-ops nuances. Your clusters will be down for a time, so make sure you understand what data sources in your environment will show gaps. For example, data polled for indexing using a script may miss an interval, any UDP data sources going to a Splunk instance you stop services on will not be indexed, etc.
3. Check that the basic services are working: LDAP logins, key apps, scripted inputs, critical data source checks, forwarder management has check-ins, license server is shows all’s well, CM show’s all’s well, etc.
4. This is a brilliant time to document the instance-specific details of your upgrade process. Include the validation checks you ran in your docs.
FYI, we've posted an upgrade roadmap with links to the latest documentation to help with upgrade planning. Check it out and let us know if you find it helpful. What's the order of operations for upgrading Splunk Enterprise?
Splunk is generally upgraded from the SH tier down. The ES staging instance implies you’re running ES on the SHC.
I have linux hosts so I’ll use a distributed shell to apply the same command across multiple nodes:
1. Grab a backup of the core Splunk configurations. You can copy the folder on each node to a new folder (ugh! And it assumes that any index buckets are NOT under %splunk_home%,) or just run diags to keep a copy.
2. Follow the upgrade instruction noted above per-tier: Untar the latest release on top of the old installation. Start services, and check for errors. Bring the tiers back up in the order suggested in the docs. NOTE: Due to special restrictions on clustered nodes, read the doc links above carefully as there are order-of-ops nuances. Your clusters will be down for a time, so make sure you understand what data sources in your environment will show gaps. For example, data polled for indexing using a script may miss an interval, any UDP data sources going to a Splunk instance you stop services on will not be indexed, etc.
3. Check that the basic services are working: LDAP logins, key apps, scripted inputs, critical data source checks, forwarder management has check-ins, license server is shows all’s well, CM show’s all’s well, etc.
4. This is a brilliant time to document the instance-specific details of your upgrade process. Include the validation checks you ran in your docs.
The latest documentation update for customers with both a SHC and an index cluster are here for 6.5.
The 6.4.4 documentation update for SHC and an index cluster is here.
An excellent answer, and I have used a similar method in the past, however the documentation states a different order:
http://docs.splunk.com/Documentation/Splunk/6.5.0/Indexer/Upgradeacluster
I've already provided some feedback on that documentation but would you mind getting it updated if your above method is the preferred method of upgrade?
Note that my feedback was that the search head cluster upgrade instructions for newer 6.x versions advise to transfer captaincy one node at a time rather than stop all search heads.
However been clear on whether the cluster master/indexers must upgrade before the search heads would be great, and preferably this should be in the documentation !
I'll reconfirm the order-of-ops and run it by the docs team.
Appreciated, thanks.