Revert to working Splunk?

[wwhitener]

Just curious. For our system, we must be able to revert to a working copy of Splunk, with all the saved searches, indexes, archived data, etc and must have a plan in place to revert if an upgrade fails for some reason. We've tried "reinstalling" from the Linux rpm and then copying over the files that we identified through the upgrade documentation and that failed as the indexes didn't carry over. That failed and we were not able to restore fully to the prior version.

Is there any documentation on how to revert in the case of a failure?

Thank you in advance!

[wwhitener · Accepted solution]

Thanks. The reinstall seems to work, but I need to do more testing.

Edited to add:

Here are the steps that I followed. I'm going from 4.0.1 to 3.4.5.

1) Run /opt/splunk/splunk diag before you do the update to 4.0.1. Save this somewhere else. I saved it to /root/Desktop.

2) Do the manual uninstall for 4.0. The rpm uninstall would successfully complete, but I had
lots of problems after that. When I did the manual uninstall, it worked. Instructions are here.

3) Install the 3.4.5 version.

4) Start splunk. I did a sanity check here and made sure that I could get in with no errors on the screen. Accept the license.

5) Stop splunk.

6) Explode the splunk-diag.tar. I ended up with a splunk-diag directory on my /root/Desktop.

7) Rename the splunk-diag to just "splunk" to make copying easier. Then copy over the installation in /opt/splunk with

\cp -rfv ./splunk/* $SPLUNK_HOME

😎 Restart.

Hey, let me know if this works for other setups. Also, this is a point-in-time reversion--whatever point in time you did the splunk diag, that's what you get.

Thanks.

[wwhitener]

OK. This didn't work on another of our test systems. So, this is definitely something to test and retest if you actually are required to have a backout procedure.

[gkanapathy]

The simplest way is simply to back up the Splunk directory completely, and simply replace it (removing/deleting the new one) if your upgrade fails. This doesn't address the data, but old data is not modified by upgrades. However, if you index new data in the new version, it may or may not be usable in an older version. (e.g., data indexed by 4.2.x is not usable in 4.1.x and down, though any old data is still usable in both versions).

[wwhitener]

I ended up with some 4.2 data in the indexes as I went through the upgrade procedure, so I think that the data got corrupted on the way through. I can restore to 4.1 without issues, but going all the way back to 3.4.5 isn't happening so far.

Is there any way to figure out what data is from the upgrade and take it out?