Migrating Data

[Kendo213]

I have 4 indexers in a cluster and need to get the data in the cold volumes migrated to a new set of storage. What would be the best way of going about this? I assume it would need to occur with one indexer at a time, but I'm not clear on how to move forward, and there isn't documentation around this.

[acharlieh]

"Best" is a function of details of your environment... Are you on Windows or Linux. If on Linux are your existing cold volume on an LVM powered lv? By "new set of storage" do you mean entirely new indexers, or new disk presented to existing indexers? Or if just existing indexers do you have any spare hardware or is it all provisioned? Is new disk SAN or physically presented? Are your existing index definitions defined in terms of volumes in indexes.conf or static paths? How much data are you needing to move (approximately, per indexer).

All of these things would feed into a number of different options that could be used to plan a successful migration, and set expectations with your users. On the plus side with an indexer cluster this is a very doable thing either by yourself or with the help of Professional Services, just a lot of details to iron out to make "best".

[Kendo213]

Redhat Linux, Splunk 6.6.4, Virtual Machines with volumes on 3 different storage tiers.

We are keeping the same Indexers, however we want to relocate the data that is on a specific volume to a new volume located on cheaper storage. The thought is to mount a temporary volume, copy the data over, and then mount that in the same path as the existing cold mount one indexer at a time. The existing volume definitions are defined on the cluster master and deployed to the indexers. We are needing to move about 1.5TB per indexer. Of course the indexer would need to be in maintenance mode, or Splunk stopped, while we're copying data around, and then started, and allowed time to catch up on all of the buckets.

[sloshburch]

Hi @kendo213. I would also recommend...

1. ...making sure to sanity check this with your Splunk account team. They may be able to provide a more sustained assistance that is more customer to your circumstances.
2. ...validate your approach with a lab environment. Never go straight to prod.
3. ...read up on volumes to make sure you're not overlooking any implementation options