I am currently planning an upgrade of our Splunk distributed infrastructure and am looking for some guidance. We currently have 1 license/deployment server, 1 search head which is also running the Enterprise Security app, 2 indexers, and 1 heavy forwarder. The version of Splunk is 5.0.5 and the version of the Enterprise Security app is 2.4.1.
The documentation mentions the need to test the currently installed apps before upgrading the live system(s). We do not have a test environment, so I am apprehensive.
Is the upgrade from 5.0.5 to 6.0.2 straightforward? Do individuals commonly experience problems when upgrading? The documentation made upgrading the Enterprise Security app seem complex. Is that really the case?
Lastly, I am curious about backups. Our indexes reside on external storage. Does it suffice just to backup the Splunk directory structure before upgrading? Is the indexed data affected in any way by the upgrade?
There are pieces of the migration that are straight forward, and there are the steps that could be challenging based upon the specifics of your Splunk Enterprise infrastructure and apps. There are excellent docs on the core Splunk Enterprise upgrades, and is sounds as though you found them just fine.
Testing is always appropriate, and relatively easy in these days of hypervisors. But to be clear, the biggest question to be answered by testing is verification that the apps work. In the case of Enterprise Security, the answer is ES 3.0 requires Splunk Enterprise 6.0 to function, so that's a yes. Figuring out what else has been installed on your search head and if it will function is an open question. Deciding that any non-ES app can be ignored until the rest of the upgrade is complete would be one way to handle the issue.
A backup of the core Splunk Enterprise folders will come in handy. The docs on ES 3.0 state the installer app will also grab a copy of the old ES installation while migrating. Upgrading form Splunk Enterprise 5.x to 6.x doesn't change the data in the indexes, but will add additional files adjacent to the indexes for storage of the accelerated data models. The Known Issues for ES 3.0 has additional information about storage use for that feature.
Ultimately, the question comes down to the priority and perception of complexity. Even the most knowledgeable Splunk administrators can run into issues during an upgrade. If there are just too many moving parts and business critical functions to feel comfortable about the whole process, contact your sales engineer or professional services for assistance with the upgrade.