How to back up and restore indexed data when upgrading our indexer clustering environment from Splunk 6.2 to 6.3?

[bkumarm]

We are running a distributed clustered Splunk environment on version 6.2
We are planning to upgrade to 6.3 due to definitive requirements.
As part of the upgrade instructions, it is mentioned to take a backup of files/indexed data and configurations.
We could not find any instructions on how to back up and restore the indexed data.
We are running on a Unix environment and have a large amount of data coming in (in TB) daily.
Any instructions on how to back up and then restore after upgrade?
Also the instructions say all Indexer peers should be stopped.

We are skeptical about this, since it will lead to loss of data. Any other alternative?

[ChrisG · Accepted solution]

There is also documentation about backing up indexed data and configurations:

• Back up indexed data in the Managing Indexers and Clusters of Indexers manual.
• Back up configuration information in the Admin Manual.

[bkumarm]

Hi Chris,
These links provide raw information and are not clear to execute.
we need steps/commands to execute and procedure to back/restore data even in case the upgrade fails.
Thanks

[ChrisG]

As the topics say, the backup operations are just copying directories in the file system. You can do that however you want, from the command-line or using a backup utility. That part depends on your environment, so it's hard to give precise steps. Manually copying the necessary directories (which the documentation specifies) will create the backup you seek.

[muebel]

Hi bkumarm, at a certain scale (daily TB) it probably makes more sense to setup offsite replication via a multi-site index cluster. Otherwise you simply copy the indexes off to some other storage system, and then copy it back as the recovery process.

Concerning the "all indexers stopped" idea, this is important for version upgrades, as there might be fundamental changes to the way intra-splunk communication works, and so the components need to be on the same version to avoid issues. Depending on the length of the downtime, and each individual forwarder thruput rate, you may or may not experience any issues with queue buildup from the forwarders. The tcpoutput queue is configurable, and you can even setup space on disk to keep events should the in-memory queue fill up.

This is known as a "persistent queue," more info can be found here http://docs.splunk.com/Documentation/Splunk/6.4.0/Data/Usepersistentqueues

Please let me know if this answers your question!

[bkumarm]

Hi Muebel,
Your explanation partly answers the approach to be taken. however the command to execute and achieve this are not available. Suppose the upgrade fails and I had to reinstall old version, how to get back the old indexed data.
there is no clear explanation of procedure (with example) on how to back up the current indexed data, how to handle the buffer data during upgrade downtime and finally how to get back the data.
I am looking for all these details.

Thanks,

[somesoni2]

This may help.
https://answers.splunk.com/answers/2291/how-to-backup-restore-splunk-db-to-new-system.html