Exception: <class 'PermissionError'>, Value: [Errno 13] Permission denied: '/opt/splunk/etc/system/local/authentication.conf.migratepreview'
Unable to restart Splunk after upgrade
What system is this? I assume this is some flavor of linux. Which distribution? How did you do the upgrade? (rpm? deb? tgz?) How was the original system installed?
Similar has happened to me for my last 2 upgrades (9.0.5 and 9.1.0.2). In both cases after the upgrade, some conf files end up with root:root ownership. Before the upgrade they were splunk:splunk.
Offending files (plus 1 .pyc file and migration.log in var/log/splunk):
etc/system/local/eventtypes.conf
etc/system/local/web-features.conf
etc/system/local/authorize.conf
Primary concern are the conf files. The upgrade is by .tgz file run as splunk:splunk. The initial start is run by root cuz it needs root permissions to create the systemd boot-start file. Is this just going to keep happening since I need to run "splunk enable boot-start . . . " as root?
It's not a big deal to run chown to fix everything, but it is a manually step when is sometimes forgotten.
Things don't happen by themselves usually. You must have sone something as root so that resulting files ended up being owned by root.
Make sure you performed the upgrade as the right user (the owner of Splunk as opposed to root). Ensure all files in $SPLUNK_HOME are owned by the user running Splunk.
I have inherited this system and the original owner is no longer with us.
How do I determine which account to use?
Is it better to uninstall and reinstall?
If Splunk is running : you can find the user running Splunk using
ps -aux | grep -i Splunk
And find which user is running the process
If Splunk is not running : you can check who owns the $SPLUNK_HOME folder (usually under /opt/splunk)
As @richgalloway mentioned, you can try this command o be sure the user that will run Splunk has the ownership of the install path :
chown -R <user_meant_to_run_splunk> /opt/splunk
Good luck!