Can you upgrade Splunk Enterprise Security on a test server that points at the same Index layer?

[chrisbennett]

I am planning out the first upgrade of Splunk Enterprise Security (Splunk ES) and am working out how. When we installed the system with Professional Services, we had a test server and our production search head pointing at the same index layer. These were both the same version of ES and allowed us to test some configs. Now that I am working on a major version upgrade (3.3.1 to 4.1.4 to 4.7.1), will it break things having a test server upgraded to 4.1.4 if the 3.3.1 search head is still up? Or is the better strategy now to snapshot the Prod server and upgrade there?

[micahkemp · Accepted solution]

One issue you'll run into by creating a new ES SH to replace your old one is the datamodel acceleration. Each SH that has DM acceleration enabled has its own set of accelerated data on the indexers. This means that you'd be doubled up on DM acceleration storage. This may or may not be OK with you, but it's certainly worth considering before you go down this route.

A potential workaround/solution would be to test your new ES SH without enabling DM acceleration until you're ready to decom the old ES SH. This may cause issues with your indexers if you have more than a few correlation searches enabled at the same time, as the searches will be more expensive to perform. Also worth noting is when you retire your old ES SH you need to look into how to force its accelerated data to be deleted from your indexers.