What are the steps to upgrade Splunk Enterprise Security on a SH cluster from 5.2.2 to 6.1.1? The Docs are confusing

[chans28]

I'm on Splunk Enterprise 8.0.5 for this question.

Upgrading ES from 5.2.2 to 6.1.1:
The Splunk docs say install 6.1.1 on the Deployer via the GUI first which will put ES 6.1.1 app in the $SPLUNK_HOME/etc/apps  directory. I'm clear here so far

Then it says choose a MODE before pushing 6.1.1 out using 

splunk apply shcluster-bundle

 Which we know will take the apps in $SPLUNK_HOME/shcluster/apps on the deployer and create a bundle to push out to the SH Members. 

So here is my question: When does the 6.1.1 I deployed using the GUI in $SPLUNK_HOME/etc/apps get copied to  $SPLUNK_HOME/shcluster/apps on the Deployer so it can be pushed out in the bundle???

Am I supposed to do that manually?

[mhessel]

no, it used to be that you had to manually copy the files over to the shcluster/apps directory, but now that is automatic. On installation of the application to the Deployer, it should detect the existing copy of Enterprise Security in %SPLUNK%/etc/shcluster/apps, and it will automatically update them as a part of the installation you run on the SplunkWeb part of the deployer. The relevant information is https://docs.splunk.com/Documentation/ES/6.2.0/Install/InstallEnterpriseSecuritySHC - look under the heading "Installing Enterprise Security in a search head cluster environment"

[chans28]

So I ended up using the command line instructions and that worked
https://docs.splunk.com/Documentation/ES/6.1.1/Install/InstallEnterpriseSecuritySHC#Installing_Splun...

The GUI instructions don't work as I expected. After installing ES 6.1.1 as an app, the merg of ../etc/apps and /etc/shcluster/apps doesn't seem to happen after running:

splunk apply shcluster-bundle -target <URI>:<management_port> -auth <username>:<password>

https://docs.splunk.com/Documentation/ES/6.1.1/Install/InstallEnterpriseSecuritySHC#Installing_Enter...

[mhessel]

just as a thought, maybe you ran into the upload limitation in the web gui?  there is a note in there near the top stating you need to increase the max_upload_size to 1024 (1GB).  If you don't do this, the install of the ES package will fail as it will terminate the upload to the Deployer early, and nothing will work then.

...

Increase the Splunk Web upload limit, for example to 1GB, by creating a file called $SPLUNK_HOME/etc/system/local/web.conf with the following stanza.

[settings]
max_upload_size = 1024

...

[chans28]

I did catch that part of the instructions too. So I did update the max_upload_size. Everything went smooth installing ES 6.1.1 as a singular app on the SH Deployer. I was able to bring it up and it looked fine. Just the next step of pushing it out to the SH Members that didn't work. 5.2.2 was still on the SH Members after the push. So the GUI way of upgrading in the docs is not clear to me. At what point during the GUI instructions does the newly installed ES 6.1.1 in ../etc/apps get merged with the old ES 5.2.2 in ../etc/shcluster/apps so it can be pushed out to the SH Memebers?

[mhessel]

I can only tell you what it does when it works 🙂

once you upload it, it prompts you to configure it, you go through the configuration as usual, and either before or after that, it needs to restart Splunk, after you restart it, you should be good to go, and the configurations should all be updated in the %SPLUNK_HOME%/etc/shcluster/apps directory.

[chans28]

So if I understand you correctly, you had an older version of ES deployed to your SH Cluster (Curious what version was the old one?). Then you went through the GUI steps and your SH Members were updated with the new version of ES while maintaining all the old config from your older version?

[mhessel]

Yes, that is correct.

I believe before running 6.1.1, I was on 6.0.2, prior to that I had 5.3.1.

This particular installation has been progressively updated since about 4.0.2 probably.  Upgrading to 5.3.0 was the point where you didn't need to stage the upgrade on another standalone and then copy the packages over, you run the installer from the deployer, and that updated the necessary packages.

[chans28]

Ok I will try the GUI instructions again but go from 5.2.2 ->5.3.x->6.1.1 just for my own curiosity but the CLI way seems to work without and adjustments.

Thx for all the feedback.

[richgalloway]

Given that this is a different process from updating other apps, it needs to be much better documented.

[richgalloway]

Whenever you find confusing docs be sure to submit feedback on that page.  Splunk's documentation team is great about updating the docs in response to feedback.

To answer your question, moving the app from $SPLUNK_HOME/etc/apps to $SPLUNK_HOME/etc/shcluster/apps is a manual process.  Do it after setting the deployment mode and before running the apply shcluster-bundle command.