Hello all. Apologies in advance if the answer to these questions are documented elsewhere, but I've not been able to find any direct answers so far.
I am about to upgrade our Deployment Server and Heavy Forwarder to our cloud instance (each role on a separate Windows server VM) from 6.2.5 to 6.5.1. I've not upgraded these before, so am a little nervous. I'd like to take VM snapshots prior to upgrading and roll back if there are issues. Is it safe to do this? Is there a risk of data duplication from the Heavy Forwarder if a roll-back occurred? There are several hundred GB worth of Syslog files that we leave on that server and I'd hate for it all to show up twice is searches (not to mention blow our daily limit).
Also, on the Deployment Server, are there any specific directories that should be backed up/copied so that UF configurations can be preserved?
If by restoring the snapshot on the HF, you'd be restoring the syslog data stored on that server at the same time, I don't believe you'd risk duplication by restoring the snapshot. You would have lost the incoming data between the time the snapshot was taken and when it was restored (but potentially already indexed by Splunk during this time, so not really "lost").
If the syslog data won't be restored to the snapshot like Splunk will, then you risk duplication on ingested logs during the timespan between when the snapshot was taken, and when you decided to roll back. To reduce this risk, you can stop splunk before taking the snapshot, and you can stop splunk prior to the restoring the snapshot and backup the fishbucket folder. Once you backup the fishbucket, you should be able to restore the snapshot, and overwrite the old fishbucket with the new one. This should keep the pointers for the syslog data at what they were before restoring the snapshot.
You won't risk complete duplication of all data, just the data between the snapshot and restore if you don't back up the fishbucket prior to restoration.
For the Deployment Server, it's safest to zip the entire etc folder. But you're probably really only going to want the /etc/deployment-apps, /etc/apps, and /etc/system/ folders backed up.