Stuck with Splunk ES Upgrade

[nareshinsvu]

Hi Helpers - Below is my usecase where I am stuck with my ES upgrade. 

My Splunk version recently upgraded from 7.2.7 to 8.1.3

Post the Splunk upgrade, Splunk ES views were throwing pop-up messages “Timelines could not be loaded”. Splunk ES was on 4.5.2 which was working fine on Splunk 7.2.7. Since it looked incompatible, we planned to upgrade it to 6.2.0. Below is the process followed.

It's on a SHC environment with 3 Search Heads

1. On ES Deployer, take backups of etc/shcluster/apps to etc/apps folders
2. On ES Deployer, copied the apps (SA-*, DA-*, SplunkEnterpriseSecuritySuite) from etc/shcluster/apps to etc/apps folder
3. Ran the upgrade command – (/opt/splunk/bin/splunk install app ./splunk-enterprise-security_620.spl -update 1)
4. Ran the essinstall command as per the install documentation – (/opt/splunk/bin/splunk search '| essinstall --deployment_type shc_deployer' -auth admin:TelstraDR01 action=upgrade) – (Output attached)
5. /opt/splunk/bin/splunk restart – (Multiple Invalid Stanzas and Output attached)
6. Planning to replace all conf files from backup apps directories to the upgraded apps directories as we have noticed there is a change in the conf files. Not sure which ones to replace and the consequences – PENDING

Bit confused with the documentation. Upgrade documentation didn't have essinstall action=upgrade part. But read about it in some blog. Am I supposed to run it or not?

When I followed the upgrade documentation, only SplunkEnterpriseSecuritySuite app folder got changed and the remaining SA-* and DA-* apps were unchanged.

But SA-* and DA-* got changed when I ran essinstall command followed by splunk restart.

All this is just on deployer. Haven't pushed any changes to search heads.

Has anyone recently did ES upgrade and can share me clear steps to be followed?

Raised a Splunk support case and they are advicing just to follow the upgrade doco which is fully not clear.

Thanks & Regards,

Naresh

[nareshinsvu]

Any help is highly appreciated