Splunk upgrade process seems to be very confusing from 7->8. I stop splunk using a systemctl splunk stop to stop the services because if i stop using the splunk user it starts again since splunk is configured to as systemd service. Edit the splunkd.service file as root as the new splunkd service file should not contain user=splunk and other commands. I use the file given by splunk here https://docs.splunk.com/Documentation/Splunk/8.0.3/Admin/RunSplunkassystemdservice I am using a rpm based install and i use: rpm -i --replacepkgs --prefix=/splunkdirectory/ splunk_package_name.rpm This is use to replace the exsisting install package of 7 and the new package is 8. This command allowed to be executed as splunk user and i need to be a root user else i get error if i run as non root user. error: can't create transaction lock on /var/lib/rpm/.rpm.lock (Permission denied) Next i start the splunk as per the upgrade recommendation from splunk sudo splunk start https://docs.splunk.com/Documentation/Splunk/8.0.3/Admin/RunSplunkassystemdservice#Upgrade_considerations_for_systemd This gives me lots of "Invalid key in stanza" while starting , also the splunk is running as a root process now -bash-4.2$ ps -ef | grep splunk root 31719 31229 0 18:08 pts/0 00:00:00 sudo su - splunk root 31721 31719 0 18:08 pts/0 00:00:00 su - splunk splunk 31722 31721 0 18:08 pts/0 00:00:00 -bash root 31806 1 8 18:09 ? 00:00:04 splunkd -p 8089 start root 31808 31806 0 18:09 ? 00:00:00 [splunkd pid=31806] splunkd -p 8089 start [process-runner] root 31834 31808 0 18:09 ? 00:00:00 mongod --dbpath=/opt/splunk/var/lib/splunk/kvstore/mongo --port=8191 --timeStampFormat=iso8601-utc --smallfiles --oplogSize=200 --keyFile=/opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key --setParameter=enableLocalhostAuthBypass=0 --replSet=A818B836-060F-4BA2-A42E-82AE5CF11FFA --sslMode=requireSSL --sslAllowInvalidHostnames --sslPEMKeyFile=/opt/splunk/etc/auth/server.pem --sslPEMKeyPassword=xxxxxxxx --sslDisabledProtocols=noTLS1_0,noTLS1_1 --sslCipherConfig=ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256 --nounixsocket --noscripting root 31938 31808 2 18:09 ? 00:00:01 /opt/splunk/bin/python -O /opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/root.py --proxied=127.0.0.1,8065,8443 root 31996 31808 0 18:10 ? 00:00:00 /opt/splunk/bin/splunkd instrument-resource-usage -p 8089 --with-kvstore splunk 32063 31722 0 18:10 pts/0 00:00:00 ps -ef splunk 32064 31722 0 18:10 pts/0 00:00:00 grep --color=auto splunk Tried to stop the splunk process and run again as user splunk, splunk process starts ok but the splunk daemon is dead systemctl status splunk ● splunkd.service - Systemd service file for Splunk, generated by 'splunk enable boot-start' Loaded: loaded (/etc/systemd/system/splunkd.service; enabled; vendor preset: disabled) Active: inactive (dead) (Result: exit-code) After starting the splunk daemon it is still in failed state, complaining about permissions. splunkd.service - Systemd service file for Splunk, generated by 'splunk enable boot-start' Loaded: loaded (/etc/systemd/system/splunkd.service; enabled; vendor preset: disabled) Active: failed (Result: start-limit) since Mon 2020-09-14 18:17:07 UTC; 871ms ago Process: 32730 ExecStart=/opt/splunk/bin/splunk _internal_launch_under_systemd (code=exited, status=4) Main PID: 32730 (code=exited, status=4) systemd[1]: splunkd.service: main process exited, code=exited, status=4/NOPERMISSION systemd[1]: Unit splunkd.service entered failed state. systemd[1]: splunkd.service failed. splunkd.service holdoff time over, scheduling restart. systemd[1]: Stopped Systemd service file for Splunk, generated by 'splunk enable boot-start'. systemd[1]: start request repeated too quickly for splunkd.service Failed to start Systemd service file for Splunk, generated by 'splunk enable boot-start'. systemd[1]: Unit splunkd.service entered failed state. systemd[1]: splunkd.service failed. Has anyone faced these same issues? Am i working in the correct order or do i need to change the order or am i missing something in between?
... View more