Installation

How to back up and restore indexed data when upgrading our indexer clustering environment from Splunk 6.2 to 6.3?

bkumarm
Contributor

We are running a distributed clustered Splunk environment on version 6.2
We are planning to upgrade to 6.3 due to definitive requirements.
As part of the upgrade instructions, it is mentioned to take a backup of files/indexed data and configurations.
We could not find any instructions on how to back up and restore the indexed data.
We are running on a Unix environment and have a large amount of data coming in (in TB) daily.
Any instructions on how to back up and then restore after upgrade?
Also the instructions say all Indexer peers should be stopped.

We are skeptical about this, since it will lead to loss of data. Any other alternative?

Labels (3)
0 Karma
1 Solution

ChrisG
Splunk Employee
Splunk Employee

There is also documentation about backing up indexed data and configurations:

View solution in original post

ChrisG
Splunk Employee
Splunk Employee

There is also documentation about backing up indexed data and configurations:

bkumarm
Contributor

Hi Chris,
These links provide raw information and are not clear to execute.
we need steps/commands to execute and procedure to back/restore data even in case the upgrade fails.
Thanks

0 Karma

ChrisG
Splunk Employee
Splunk Employee

As the topics say, the backup operations are just copying directories in the file system. You can do that however you want, from the command-line or using a backup utility. That part depends on your environment, so it's hard to give precise steps. Manually copying the necessary directories (which the documentation specifies) will create the backup you seek.

muebel
SplunkTrust
SplunkTrust

Hi bkumarm, at a certain scale (daily TB) it probably makes more sense to setup offsite replication via a multi-site index cluster. Otherwise you simply copy the indexes off to some other storage system, and then copy it back as the recovery process.

Concerning the "all indexers stopped" idea, this is important for version upgrades, as there might be fundamental changes to the way intra-splunk communication works, and so the components need to be on the same version to avoid issues. Depending on the length of the downtime, and each individual forwarder thruput rate, you may or may not experience any issues with queue buildup from the forwarders. The tcpoutput queue is configurable, and you can even setup space on disk to keep events should the in-memory queue fill up.

This is known as a "persistent queue," more info can be found here http://docs.splunk.com/Documentation/Splunk/6.4.0/Data/Usepersistentqueues

Please let me know if this answers your question!

bkumarm
Contributor

Hi Muebel,
Your explanation partly answers the approach to be taken. however the command to execute and achieve this are not available. Suppose the upgrade fails and I had to reinstall old version, how to get back the old indexed data.
there is no clear explanation of procedure (with example) on how to back up the current indexed data, how to handle the buffer data during upgrade downtime and finally how to get back the data.
I am looking for all these details.

Thanks,

Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

Industry Solutions for Supply Chain and OT, Amazon Use Cases, Plus More New Articles ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Enterprise Security Content Update (ESCU) | New Releases

In November, the Splunk Threat Research Team had one release of new security content via the Enterprise ...