How to migrate data of cold and thawed path to different location?

[klowk]

Hi all,

I have following situation.

We had an indexer cluster with 4 peers were is currently still enough storage on our SSD's, so the home, cold and thawed path is for all the same(/data/<index_name>/(colddb|db|thaweddb)). Now we will extend the storage with HDD and plan to migrate the cold and thawed path for all indexes to a different storage location(/archive/<index_name>/(colddb|thaweddb)).

Now is the question how should it work ?

I want to minimize the downtime so i would prepare the new locations on all 4 indexer peers and would already do a copy of all buckets in cold and thaweddb to the new location. Now the question can i reduce the bucket roll activity with starting maintenance mode?! So i would activate the maintenance mode make again a copy to get all bucket files on the same state, now i would adjust the indexes.conf and initiate a rolling restart afterwards i would disable the maintenance mode. But does it work to make a rolling restart when maintenance mode is active?

Or do i only have to copy the files to new location change indexes.conf and restart, but what is if a bucket roll take place from warm to cold, can i copy the files again from old to new directory and restart again, because i do not want to loose any data.

please give me an advice because i did not find any information in the documentation when and how to restart the indexer cluster in such case.

kind regards
Kathrin

[isoutamo]

Hi

Disclaimer: do it your own risk or ask Splunk professional service help 😉

As maintenance mode don't forbidden the bucket movements inside local node (just stuff between nodes) it didn't help you to prevent bucket movement from warm to cold and frozen.

I think that the best option is to have a maintenance break when you can move buckets/directories from one place to another. Basically you could do it quite quickly if/when you are first replicate those buckets with rsync on all nodes. Then on switch over time just do it last time with remove option (to get ride duplicate/frozen in colddb buckets). Probably you should also add some symbolic links there as you still have old indexes.conf file on cluster and you cannot push a new with correct paths before cluster is up?

Another option is use local indexes.conf on those nodes one by one and of course maintenance mode on CM with enough long timeouts. Stop individual instance, move/sync needed colddb and thaweddb to the new places and update local indexes.conf to point those. Then bring it up and check that it works. This for all nodes. After all has updated, disable maintenance mode and push a new indexes.conf to all nodes. Then again through maintenance mode remove local indexes.conf one by one and after this has done rolling restart and check that everything is working.

If I recall right we have done this 2nd option to move all internal indexes on multisite cluster from /opt to separate splunk volume without any issues.

BTW: start to use splunk volumes if you don't use those yet. Also LVM on linux is (almost) mandatory.

r. Ismo