Is it possible to rsync an indexer's colddb which resides on an NFS export from that network location to another network location while the indexer is still running the Splunk service? A different machine will be performing the rsync. I was planning on initiating rsync for the first pass while the indexer was still running. Once the rsync job was complete, stop the indexer Splunk service, then run one more rsync for incremental changes. After that point the indexer to the new colddb and start Splunk service on the indexer.
Anyone know if this is a viable option for migration of the colddb?
Yes if you needed to move the location of the colddb, that should work fine. When you bring up the new colddb location, your indexer may re-freeze buckets that froze between the start of the rsync and the completion. If you don't have a cold2frozen action set, then you should have no problem; the indexer will just remove the bucket. If you do have a freeze location or a cold2frozen script, you might want to verify which buckets were already frozen and remove them before you bring the indexer back up. Searching the indexer logs for bucketmover events to identify the buckets that were already frozen.
Thanks Jeremiah. I believe the second rsync job should get both colddb location exactly in sync, as it will be run once the initial rsync job is complete and only after the Splunk service is stopped on the indexer. Also, if there are some buckets that should be frozen, that should be fine, since our frozen action is to delete those buckets.