Hello. Our site does not currently use or have a particular need for indexer clustering. We are, however, about to shut down the datacenter that holds our indexers and move that functionality to a different datacenter a couple thousand miles away. The indexers themselves are not moving. We have a fairly steady stream of events flowing into those indexers currently.
We are faced with the problem of how to get existing indexed events to the new datacenter without dropping events coming from forwarders. We've got 100's of GB of data in the indexers now so a directly copy/rsync would take a very long time and we can't shut the indexers down while that happens as events from forwarders would be dropped. I realize we can copy the cold buckets directly, but even that will take a long time to do directly.
I was wondering if using indexer clustering (multisite) might be a way to get Splunk to do the replication for me. If I could setup a cluster that replicated all the data in the datacenter that's going away to the new datacenter, wait until all the data is replicated, then remove the old datacenter's indexers from the cluster and shut them down, is that a viable strategy?
No. Pre-clustering-buckets will not get replicated by enabling clustering, so unless your existing datacenter already is a clustered splunk install you won't get any benefit from clustering.
Instead, do this:
- Copy old buckets, warm and cold, to new indexers
- Switch forwarders to new indexers
- Restart old indexers to force a hot-bucket-roll
- Copy new warm buckets to new indexers
This will not drop any data from forwarders and move all indexed data over. Yes, copying will take a while - but that issue applies to any approach. If your WAN link is very slow, consider mailing or carrying (encrypted!) external HDDs to move the bulk.
OK, thanks. I had actually done that method once for another move long ago. I was hoping it might be easier.
I must agree with Martin here. Even if it takes "some time", doing an rsync of cold buckets is a decent strategy. You rsync several times, avoiding hot buckets on purpose. Each rsync brings your destination site "closer in time" until you're ready to flash cut. You then shut down Splunk at site A, do a final quick rsync, reconfigure your forwarders to point to new indexers, and bring it up at site B.
There are "ways" that professional services can take existing buckets and make them clusterable ; however, this is not entirely trivial, especially when we are speaking of multi-site clustering. And, when it was done, you would need to keep things "in cluster mode" forever, so there could be some gotchas.