If I had a single-instance Splunk Enterpirse instance that I was decommissioning for a new instance, and I was unable to do a migration-and-overwrite procedure, I'd need to hand-review and copy all of my configurations and data between the old and new instances. That's a good bit of work, as you're rebuilding the infrastructure.
Once the new instance was properly configured with basics such as licensing and was running, I'd move over the user configurations ($splunk_home/etc/users) and auth settings. Have a non-admin user test the auth basics.
Once auth was tested and working, I'd make a list and reinstall all the "enabled" apps from the old instance on the new one.
If it's not a splunkbase app, it might be custom made. So you'll migrate those enabled, unique apps by hand from the old instance to the new one. Once again, have a non-admin user login and test the basics.
Get a diag from the old instance for configuration backup and verification purposes.
I'd then want to manage the index data. But first I'd review the volume/index settings on the old and new instances to be sure the new instance has the same or more disk space. Decide if you need a new indexes.conf or if you can use the old one. But you should very carefully review the indexes.conf and splunk-launch.conf for unique volume or index paths (default is $splunk_home/var/lib/splunk/, but that doesn't mean that's what you've used) that might not exist on the new instance. Get the new instance indexes.conf configured and test that the indexes are made on disk as desired.
As new data is always coming in, you have a choice to make at this point: flip the forwarders over to the new instance first, or migrate the old instance data first? Most will flip the forwarders over for data continuity and then migrate the old data after.
Flip the forwarders and other data sources to the new node. Test that data is going to the right indexes.
Migrate the old data. If you still have single-digit bucket numbers in the old data, you should give the bucket numbering a bump because you don't want to overwrite your newly indexed data with old stuff.
Once that's accomplished, you can invite more users on to the new instance. Test all the apps, make sure you can see old and new data, and prepare for users saying they're missing some configuration.
I've discussed a single-instance by-hand migration, but it gets significantly more complex for a distributed environment. This is where professional services is often called-in for guidance, procedural review, and order-of-operations confirmation.
... View more
