topic Re: Why is Splunk not starting after upgrade to 8? in Installation

Why is Splunk not starting after upgrade to 8?

48tfhd86gv — Tue, 05 Nov 2019 14:02:12 GMT

Went to upgrade splunk-7.3.0-657388c7a488-linux-2.6-amd64 to splunk-8.0.0-1357bef0a7f6-linux-2.6-amd64.deb on a standalone instance of Splunk running on Debian 10.1.

The install process went fine. The deb file recognised there was an existing vesion and shut it down before continuing. however now v8 just won't start.

Looking at journalctl -xe, I'm seeing the following :

Nov 05 13:56:35  splunk[597]: Couldn't change ownership for /sys/fs/cgroup/cpu//system.slice/Splunkd.service: Operation not permitted                     
Nov 05 13:56:35  splunk[597]: Error perform systemd preparation: Operation not permitted                                                                  
Nov 05 13:56:35  systemd[1]: Splunkd.service: Main process exited, code=exited, status=1/FAILURE

The cited file /sys/fs/cgroup/cpu//system.slice/Splunkd.service does not exist ? Is this something the installer should have created ?

Re: Why is Splunk not starting after upgrade to 8?

ivanreis — Wed, 06 Nov 2019 04:28:54 GMT

Per the error message it seems to be a permission issue. The splunk process is trying to load, but the Debian is not allowing to start. Check which account was created to run the splunkd service. I believe you have the redeploy the ownership the user created to run splunkd service.

Re: Why is Splunk not starting after upgrade to 8?

48tfhd86gv — Wed, 06 Nov 2019 05:53:54 GMT

@Ivanreis, but the error says "Couldn't change ownership for /sys/fs/cgroup/cpu//system.slice/Splunkd.service", but "/sys/fs/cgroup/cpu//system.slice/Splunkd.service" does not even exist ???

Re: Why is Splunk not starting after upgrade to 8?

ivanreis — Wed, 06 Nov 2019 06:40:55 GMT

I mean that it is possible the account that is being used to start the splunk service does not have all the required access to start splunkd service, so this process splunkd should run successfully, so if for some reason the user is not able to start this service, it is potential problem. Use the root user to try to redploy the permission for the user you have created to start splunk.
If it did not work, try to reinstall the previous version and after that redeploy Splunk 8 again.

Re: Why is Splunk not starting after upgrade to 8?

DavidHourani — Wed, 06 Nov 2019 07:40:08 GMT

Hi @48tfhd86gv,

From "Error perform systemd preparation: Operation not permitted " Seems like the Splunk user is trying to create Splunkd.service and doesn't have the permissions. Try starting Splunk with the root user then reverting the file ownership back to the splunk user.

Re: Why is Splunk not starting after upgrade to 8?

48tfhd86gv — Wed, 06 Nov 2019 16:47:11 GMT

@DavidHourani Perfect, that worked. Thank you for saving me from the nightmare of having to re-install !

Re: Why is Splunk not starting after upgrade to 8?

DavidHourani — Wed, 06 Nov 2019 17:10:09 GMT

You're welcome! Please accept the answer if it was helpful 🙂

Re: Why is Splunk not starting after upgrade to 8?

sarweins — Wed, 06 Nov 2019 21:55:40 GMT

I would also make sure everything on your instance is compatible with 8.0

Re: Why is Splunk not starting after upgrade to 8?

yaunj — Thu, 07 Nov 2019 11:17:10 GMT

We had some issues running Splunk 8 as a non-privileged user (i.e. not root).

The directories /sys/fs/cgroup/cpu/system.slice/Splunkd.service and /sys/fs/cgroup/memory/system.slice/Splunkd.service are created when the Splunkd.service is started by systemd. They are used to influence the cgroup that the Splunk service is running under, and can be used to control CPU and Memory usage by the service.

In previous versions of Splunk, the systemd unit file generated by splunk enable boot-start -user splunk contained two ExecStartPost statements that changed the permissions for the cgroup directories. It also set User=splunk in the unit file. You can still see remains from this in the online documentation.

With Splunk 8, it seems that the Splunk process itself tries to manipulate these directories. When started with User=splunk (or a different non-privileged user), this is denied by the system, as these folders are owned by the root user. This also happens so soon in the startup process that Splunk terminates before the ExecStartPost hooks are run by systemd. If started without the User= statement in the unit file, Splunk starts as root, and drops privileges to SPLUNK_OS_USER from ${SPLUNK_HOME}/etc/splunk-launch.conf.

There are two options on how to fix this issue if you have been running Splunk as a non-privileged user via systemd:

Don't use User=splunk in the unit file (and also drop the ExecStartPost statements). As long as SPLUNK_OS_USER=splunk in ${SPLUNK_HOME}/etc/splunk-launch.conf, Splunk will fix the permissions and drop privileges.
Change the ExecStartPost statements to ExecStartPre. This ensures that systemd changes the permissions before Splunk is actually started.

Re: Why is Splunk not starting after upgrade to 8?

48tfhd86gv — Thu, 07 Nov 2019 19:03:21 GMT

Thanks for the great insight @yaunj . I actually think your answer should be the accepted one as it provides much better explanation and background detail.

I have to say, that based on this and prior experiences, although Splunk is a great product, some aspects of their documentation and software development are of questionable quality.

Re: Why is Splunk not starting after upgrade to 8?

DavidHourani — Thu, 07 Nov 2019 19:08:50 GMT

yeah totally agree with @48tfhd86gv, nice work @yaunj

Re: Why is Splunk not starting after upgrade to 8?

aoleske — Tue, 11 Feb 2020 16:14:16 GMT

Thanks! this solved my issue also...

Re: Why is Splunk not starting after upgrade to 8?

nickhills — Tue, 11 Feb 2020 16:19:18 GMT

I wonder if running splunk disable boot-start followed by splunk enable boot-start -user splunk following the upgrade would repair the pre 8.x unit files?

Re: Why is Splunk not starting after upgrade to 8?

Spranta — Tue, 29 Sep 2020 07:01:00 GMT

Great Job!

Re: Why is Splunk not starting after upgrade to 8?

MichaelPolla — Fri, 30 Apr 2021 06:43:31 GMT

Awesome, thanks. 😊

It worked. Note that in my case I also changed for SystemD :

splunk enable boot-start -systemd-managed 1 -user splunk

PS : I didn't find how to format the code like the others (inline, red text and pink background) 😪 that's the best I could do...