Deployment Architecture

How to move index from one hard drive to another in Splunk clustered environment?

Champion

Hi everyone,
I am working on Splunk clustered environment, where i have 3 indexers,1 search head, and 1 head.
Now i am facing the problem with low disk space. i configured index in index.conf in head node.
my index is replicating in 3 indexers. if want to change index path in every indexer. what should i do please help me on this. now i have given path D://splunkdb/indexname. my index resides on D drive in every indexer. now i want to move this to F: drive in each indexer.

————————————
If this helps, give a like below.
Tags (2)

SplunkTrust
SplunkTrust

I would not suggest making any changes on a Indexer locally. Here is my suggestion:

Lets assume that the original Index is at /opt/splunk/var/lib/splunk/defaultdb, and the new location will be at /splunk/defaultdb.

In order to limit the down-time of each indexer to the minimum, we will do it in a few steps. First, while the service is still running, rsync the data from the old location to the new one:

Note: When running the rsync command, use a trailing slash only after the source path, and not after the destination path.

  1. rsync -auv /opt/splunk/var/lib/splunk/defaultdb/ /splunk/defaultdb.
    This will create an initial copy of the data to the new location. The initial sync may take some time, depending on the size of your data, there also may be a lot of changes to the data when that process is done due to bucket rolling from hot/warm/cold.

  2. Now we want to do another rsync to send the recent changes, this will be a lot faster. But now we will add the --delete argument, so it deletes rolled buckets from the new location. like so:
    rsync -auvv --delete /opt/splunk/var/lib/splunk/defaultdb/ /splunk/defaultdb
    if you want to be able to look at what rsync did, you can send output to a log file by adding the --log-file=/tmp/rsync-`date %s`.out to the command.

  3. Put the CM in maintenance mode and stop Splunk, and do the final sync.
    On the master:
    $SPLUNK_HOME/bin/splunk enable maintenance-mode --answer-yes
    On the Indexer:
    $SPLUNK_HOME/bin/splunk stop

  4. Do the final sync:
    rsync -auvv --delete /opt/splunk/var/lib/splunk/defaultdb/ /splunk/defaultdb

  5. Move away the old index data to a backup location:
    mv /opt/splunk/var/lib/splunk/defaultdb /opt/splunk/var/lib/splunk/OLD.defaultdb

  6. Latstly, create a symlink from the new location to the old one:
    ln -s /splunk/defaultdb /opt/splunk/var/lib/splunk/defaultdb.
    Now you can start splunk, and not have to mess around with indexes.conf

  7. After you start the indexer, make sure that the buckets are visible by check the RF/SF on the CM, after that you can take the cluster out of maintenance mode, to fill in the few buckets that been rolled while that indexer was down.
    $SPLUNK_HOME/bin/splunk disable maintenance-mode --answer-yes

Repeat the same process 1-7 on each indexer in the cluster, one indexer at a time.
After doing this on all indexers, you can change the path in indexes.conf on the master, and push out the new bundle.
A Indexer restart is required when changing the path of a index, so the master will initiate a restart.

After you did all that, and have confirmed that all the buckets are visible in Splunk, you can remove the old data, and delete the symlink:
rm -rf /opt/splunk/var/lib/splunk/OLD.defaultdb
rm /opt/splunk/var/lib/splunk/defaultdb

Please comment if I missed something.

I hope this helps.

SplunkTrust
SplunkTrust

Thanks a lot. Minor typo - step 7 should be "disable maintenance-mode", not enable.
Is there a reason you disable and enable maintenance-mode between every Indexer change? Can't you keep it on and change them all and then disable?

0 Karma

SplunkTrust
SplunkTrust

Thanks for making me aware of the typo. Fixed.

The reason for doing it one at a time, is to minimize the downtime, and a massive cluster-wide bucket fix-up. Taking your time moving them over ensures that the replication factor will remain reasonable and searches will be able to continue, since everyone has a different RF/SF setup.

Enabling and disabling maintenance mode will fix up missing buckets after the data move.

SplunkTrust
SplunkTrust

Hi,

To migrate indexes in a cluster configuration, you can proceed as follows:

  • First, verify and ensure your cluster is currently synchronized, check the master cluster dashboard

TO MIGRATE SPECIFIC INDEXES: (modification of indexes.conf)

For each peer node of your cluster, one by one, migrate your data:

  • Stop the peer node
  • Migrate your index data from the current location to your new location
  • Edit the configuration in $SPLUNK_HOME/etc/slave-apps//[defaut or local]/indexes.conf to match the new location
  • Start the peer node
  • Wait for the cluster to synchronized before operating the next peer node

In master node:

  • Edit the bundle configuration in $SPLUNK_HOME/etc/master-apps//[defaut or local]/indexes.conf to match the new location
  • Apply the bundle configuration:

    $SPLUNK_HOME/bin/splunk apply cluster-bundle

  • This will achieve a reload of the cluster without restart, in splunkd.log of master node you will see a message like:

    INFO CMMaster - All peers have reloaded the bundle without a restart

  • Verify your cluster is fully synchronized in master dashboard

  • Verify your data

TO MIGRATE ALL INDEXES IN PEER NODES MODIFYING THE $SPLUNK_DB:

For each peer node of your cluster, one by one, migrate your data:

  • Stop the peer node
  • Migrate all Splunk indexes data from the current location (default in $SPLUNK_HOME/var/lib/splunk) to your new location
  • Edit the starting configuration in $SPLUNK_HOME/etc/splunk-launch.conf to match the new global location
  • Start the peer node
  • Wait for the cluster to synchronized before operating the next peer node

The global modification of $SPLUNK_DB is easier as you don't have to alter the bundle configuration, but this does not allow to selectively migrate data by index.

These operations are not destructive but should be carefully proceeded, if possible they should be qualified in testing environment as for any configuration change, and off course you should have up to date backup of your data.

Regards,

Guilhem

Communicator

Thank you for this well written answer.
We are going to try this soon on our cluster, since we will separate cold from hot/warm buckets on different partitions and we want to keep the cluster active during the move. (I mean that is why we have clusters right?)
I will comment here afterwards to tell you how it went.

0 Karma

SplunkTrust
SplunkTrust

Hi,

You can move all indexes from one place to another by changing the SPLUNK_DB env variable in splunk-launch.conf, see:

http://docs.splunk.com/Documentation/Splunk/6.1.3/Indexer/Moveanindex

If you want to change only the location of a specific index, you need to modify the bundle configuration from your master node and apply it to your peers

0 Karma

SplunkTrust
SplunkTrust

See my new answer

0 Karma

Champion

if i change the bundle configuration from master node,what about the data indexed already.

————————————
If this helps, give a like below.
0 Karma

Champion

yes exactly..i have clustered environment.

————————————
If this helps, give a like below.
0 Karma

Splunk Employee
Splunk Employee

You're using some mixed terminology here. Can you clarify whether you have a clustered index environment?

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!