Working on migrating from a RHEL 6 VM running splunk 8.0.5 to a RHEL 8 VM with splunk latest 8.2.6 (no clustering) Read and followed the installation and migration docs and I've been able to test with some old data that its working. But another thing I'd like to do is optimize the indexes better as well and put them on new VM disks and distribute between hot/warm and cold/frozen, the problem is our indexes are pretty big.
My understanding is that in order to move/migrate the indexes, I'll need to stop splunk on the old host and copy/rsync the directories over, then modify indexes on the new host and start splunk on it (of course with the required DNS pointing and forwarders reconfig to point to the new host). But the volumes I have are about 3TB, 3TB and a large one 25TB. I tested rsync with some data directories and it looks like it would take several days. For the large volume I don't think I have any other choice but to remove the disks from the old VM and attach to the new one. But even for the smaller 2 volumes it looks like it will take almost 24 hours to copy over 5-6 TB and I don't think I can keep splunk stopped for that long, it would definitely loose data.
Am I understanding this correctly or is there a better and/or quicker way to do this? The reason I wanted to use new VM disks is because the old host has several VM disks combined to make each of the OS volumes and its just messy (e.g. the 25TB mount point has 3 underlying disks) plus with new disks I can also distribute the indexes and hot/cold buckets better between fast and not-so-fast storage.
Would really appreciate if anyone can provide any suggestions/advice.
... View more