Help upgrading Splunk indexer cluster from 6.4 to latest version of Splunk 6.5.2?

[thambisetty_bal]

Hi Splunkers,

I would need to upgrade my cluster from 6.4 to 6.5.2 latest release. I have components listed below,

1. License Master (already upgraded to 6.5.2 and using no enforcement key)
2. Cluster Master ( running on 6.4)
3. Deployment Server (running on 6.4)
4. Two Search Heads ( running on 6.4 but not in search head cluster or search head pooling. one is for adhoc and another one is for Enterprise security)
5. Six Indexers ( running on 6.4 and are in cluster and sf=1 and rf=1)

I need to upgrade all Splunk infra to 6.5.2 ( latest release) .

I read two documents saying two different procedures :

1. http://docs.splunk.com/Documentation/Splunk/6.5.1/Indexer/Upgradeacluster
2. http://docs.splunk.com/Documentation/Splunk/6.5.2/Installation/UpgradeyourdistributedSplunkEnterpris...

first document says all Indexers should be stopped and upgraded at the same time:

Upgrade a 6.x indexer cluster to a later version of 6.x
When you upgrade a 6.x indexer cluster, such as 6.2, to a later 6.x cluster, such as 6.3 or 6.4, you must take down all cluster nodes. You cannot perform a rolling, online upgrade. If you have a multisite cluster, however, you can upgrade one site at a time. See Site-by-site upgrade for multisite indexer clusters.

Perform the following steps:

1. Stop the master.

2. Stop all the peers and search heads.

When bringing down the peers, use the splunk stop command, not splunk offline.

1. Upgrade the master node, following the normal procedure for any Splunk Enterprise upgrade, as described in How to upgrade Splunk Enterprise in the Installation Manual. Do not upgrade the peers yet.

2. Start the master, accepting all prompts, if it is not already running.

3. Run splunk enable maintenance-mode on the master. To confirm that the master is in maintenance mode, run splunk show maintenance-mode. This step prevents unnecessary bucket fix-ups. See Use maintenance mode.

4. Upgrade the peer nodes and search heads, following the normal procedure for any Splunk Enterprise upgrade, as described in How to upgrade Splunk Enterprise in the Installation Manual.

If the search heads in the indexer cluster are members of a search head cluster, see Upgrade a search head cluster.

1. Start the peer nodes and search heads, if they are not already running.

2. Run splunk disable maintenance-mode on the master. To confirm that the master is not in maintenance mode, run splunk show maintenance-mode.

You can view the master dashboard to verify that all cluster nodes are up and running.

second document says :

Upgrade a distributed environment with multiple indexers and non-pooled search heads
This procedure upgrades the search head tier, then the indexing tier, to maintain availability.

Prepare the upgrade
Confirm that any apps that the non-pooled search heads use will work on the upgraded version of Splunk, as described in "Test your apps prior to the upgrade" in this topic.
(Optional) If you use a deployment server in your environment, disable it temporarily. This prevents the server from distributing invalid configurations to your other components.
(Optional) Upgrade the deployment server, but do not restart it.
Upgrade the search heads
Disable one of the search heads.
Upgrade the search head. Do not let it restart.
After you upgrade the search head, place the confirmed working apps into the $SPLUNK_HOME/etc/apps directory of the search head.
Re-enable and restart the search head.
Test apps on the search head for operation and functionality.
If there are no problems with the search head, then disable and upgrade the remaining search heads, one by one. Repeat this step until you have reached the last search head in your environment.
(Optional) Test each search head for operation and functionality after you bring it up.
After you upgrade the last search head, test all of the search heads for operation and functionality.
Upgrade the indexers
Disable and upgrade the indexers, one by one. You can restart the indexers immediately after you upgrade them.
Test search heads to ensure that they find data across all indexers.
After you upgrade all indexers, restart your deployment server.

kindly let me which document should I need to follow to upgrade my full Splunk infra.

Thanks in advance.

[adonio]

Upgrade according to docs there are 2 options:
1. all at once
2. each tier at a time
read here:
http://docs.splunk.com/Documentation/Splunk/6.6.2/Indexer/Upgradeacluster
my suggestion:
first all the management instances, e.g. License Master, DMC, Cluster Master, Deployment Server.
then upgrade the search tier, e.g . the Search Heads.
lastly, upgrade the Indexer tier, e.g. the Indexers.
stop all of them -> enable maintenance mode on Cluster Master (you already upgraded this dude) -> watch the Indexers come to life again -> disable maintenance mode -> wait for the next great Splunk release
good luck

[skalliger]

Hi,

this indeed is a little bit confusing but also an interesting setup.

Your second url provides information for upgrading a distributed environment. Which means you could upgrade indexers one by one, if they were not clustered. So, here is what you should do, according to these facts:

• You have an indexer cluster. Although it's very confusing how you have set up your indexer cluster. With a search factor and rep factor of 1, your indexers don't really act as a cluster because your data is not being replicated, but still your Search Heads will send their searches to all of the indexers and only get returned one result at the time.
• You do not have a search head cluster.
• But you do have the Enterprse Security app.

In my opinion you have two options, because you can not remain availability due to the indexer cluster, which always needs to be upgraded as a whole in a major release upgrade.
Also, search heads must have the same or a higher maintenance level than indexers: http://docs.splunk.com/Documentation/Splunk/6.5.2/DistSearch/Distsearchsystemrequirements

This means, after upgrading your master and deployment server (also interesting is, that you're using it without having a SH cluster), here are the two options. Please check the Enterprise Security system requirements first before you proceed: http://docs.splunk.com/Documentation/ES/4.1.3/Install/DeploymentPlanning, depending on your current ES app version, it might not be compatible with Splunk Enterprise 6.5 anymore (e.g. with ES 4.1.1).

(0. Please don't forget to backup your systems.)
1. You should follow the first documentation which suggests how to update an indexer cluster. This means upgrading all indexers at once as well as your search heads.
2. Upgrade your Search Heads one by one and after that, upgrade the indexer cluster as a whole. I would prefer this method because 6.5 SHs are compatible with 6.4 indexers. However, I am not sure about this when it comes to Enterprise Security.

Skalli

[thambisetty_bal]

Thanks Skalli for your response.

I could find below URL which gives benefit from indexer clustering even there is no index replication.

http://docs.splunk.com/Documentation/Splunk/6.5.1/Indexer/Clustersinscaledoutdeployments

[skalliger]

In your position, I would still open a support case to verify what you are going to do, just in case no Splunk employee answers this post.

Regarding the indexer cluster - of course you can use it just for scaling. The problem is, when one indexer is dead, your data may be lost if you have no working backup to recover from. But I assume you use some virtualised solution, so a complete data loss may be very unlikely. However, I wouldn't count on that depending on your solution (e.g. ESXi local storage vs. central storage).

Keep us updated if you contact the support! 🙂

Skalli

[thambisetty_bal]

I have already opened a case with Splunk and working on the same.

Thanks for your quick response and feedback

[Sayanta_Basak_I]

@thambisetty_balaji

Hello,

I am going ahead with a similar upgrade. Did you figure out which is the correct approach that needs to be taken for this.

Regards