Upgrade Splunk Enterprise 8.1.x to 9.2

[Na_Kang_Lim]

Hi,

As the title suggests, I want to "revive" a Splunk lab in our company, which was running on version 8.1.2.

My target is version 9.2.10

The lab consists of a full Splunk deployment - it has Cluster Master, multisite Indexer Cluster, Search Head Cluster, SH Deployer, Deployment Server and a few UFs.

I have read through some docs about upgrading Splunk, and it appeared to me that I have to through 2 upgrade steps: From 8.1.2 to 9.0 (I am planning to upgrade it to 9.0.4 first), then from 9.0.4 to 9.2.10.

Currently Splunk is running in the /home/splunk folder.

What I concern most is: How to retain the data of the Indexers?

I saw a vid showing the upgrade process, in which they tar the whole $SPLUNK_HOME folder for backup. But that process is quite challenging if you have TBs of data in the $SPLUNK_HOME/var/lib folder, right? Is there any other way to retain data, after upgrading?

Also, is there any other thing I should take note of? Any suggestions, recommendations, is welcome

[PickleRick]

You don't _have to_ do a backup just to do the upgrade. Backup is... for backup. In case something goes south you can restore your data. It's often not convenient to backup whole installation including indexed data but it's always a sound idea to backup Splunk's configuration, apps, state (kvstore, input checkpoints and so on). It's way easier to do if you have your space partitioned - store Splunk data separately from Splunk's main directory. At least for the indexers. Which you apparently don't do. So you might get away with skipping your index directories from backup but since everything by default resides in $SPLUNK_HOME/var/lib it will be tricky to properly select the stuff to backup and in case of disaster properly remove old stuff without touching the indexed data.

[Na_Kang_Lim]

If the servers are VMs, and I want to change the OS of the servers running Splunk too (like from CentOS to Ubuntu), what steps do I have to take?

[PickleRick]

You don't "change OS". It's effectively a completely new system installation. So you have to install a new OS, install a new instance of UF and - depending on your setup - either bring back your config backed up prior to destroying old setup or push a minimal config allowing you to receive config from CM, SHCD, DS...

Still, if it's a forwarder, especially HF, you're facing additional challenges with migrating inputs' states.

So tread lightly.

[verbal_666]

For my experience you have to dump all Splunk paths with a simple tar (use J or j or z compression as you want).

tar -czf splunk_$HOSTNAME.tgz (<splunk_path> es./opt/splunk)

+IDXs: move/compress Indexers "external" db paths to new Indexers to keep dbs if out of the <splunk_path=var/lib/splunk> ... i have many dbs in outer fs, like lv /splunkdb/ ... move/tar all "/splunkdb" to new machines...

for every node... move tgz to new machines... change & fix conf in .../etc/system/local/ for new hostnames, ips, etc... change any outputs.conf you use to use new Indexers.

Check also Deployment if contains some outputs.conf or fixed call to indexers or other piece of old infra, and update it for new infra. Also UFs need to be checked, if the point to fixed DS, any point to DS must be changed with new DS hostname/ip.

When you have fixed all confs (also peers in SearchHeads) to new Infrastructure, launch a new run for every node, and you should have you env ready as before 👍

Best way is to change OS, but keep all hostnames/ip, so you don't need to change point call in confs, but use the old ones.

[PrewinThomas]

@Na_Kang_Lim 

You don’t need to back up terabytes of raw index data to upgrade Splunk. As long as you perform a rolling upgrade of your indexer cluster and don’t delete or overwrite $SPLUNK_DB your indexed data will remain intact. But back up configurations.

Indexed data lives under $SPLUNK_DB. You don’t need to tar this entire directory unless you want a full disaster-recovery backup.

Backup your configs from etc.
$SPLUNK_HOME/etc → all configs, apps etc..

During an in-place upgrade, Splunk preserves buckets. As long as you don’t wipe $SPLUNK_DB, your TBs of data remain usable.

Upgrade Path
-Upgrade from 8.1.2 → 9.0.x
-Upgrade from 9.0.x → 9.2.10

Better to perform rolling upgrade.
#https://help.splunk.com/en/data-management/manage-splunk-enterprise-indexers/9.2/deploy-the-indexer-...

Check this doc before upgrade
#https://help.splunk.com/en/splunk-enterprise/administer/install-and-upgrade/9.2/upgrade-or-migrate-s...

Regards,
Prewin
If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks!

[verbal_666]

---

Upgrade Path
-Upgrade from 8.1.2 → 9.0.x
-Upgrade from 9.0.x → 9.2.10

---

9.0.x are no longer present on Splunk repository.

It's possible to upgrade,

-Upgrade from 8.x.x → 9.1.x [if there's a SH Cluster, follow how to upgrade KVSTORE/MONGODB]
-Upgrade from 9.1.x → 9.4.x
-Upgrade from 9.4.x → 10.x.x

[Na_Kang_Lim]

Upgrade Path
-Upgrade from 8.1.2 → 9.0.x
-Upgrade from 9.0.x → 9.2.10

Can I do this from instance to instance?

Like:

Do I upgrade to 9.2 from instance to instance

OR

I upgrade to 9.0 on all instances, then upgrade to 9.2 on all instances

[PickleRick]

No. You have to first bring your whole environment to the supported configuration (like 8.1->9.0), then go for another step.