Installation

Why is Splunk not starting after upgrade to 8?

48tfhd86gv
Explorer

Went to upgrade splunk-7.3.0-657388c7a488-linux-2.6-amd64 to splunk-8.0.0-1357bef0a7f6-linux-2.6-amd64.deb on a standalone instance of Splunk running on Debian 10.1.

The install process went fine. The deb file recognised there was an existing vesion and shut it down before continuing. however now v8 just won't start.

Looking at journalctl -xe, I'm seeing the following :

Nov 05 13:56:35  splunk[597]: Couldn't change ownership for /sys/fs/cgroup/cpu//system.slice/Splunkd.service: Operation not permitted                     
Nov 05 13:56:35  splunk[597]: Error perform systemd preparation: Operation not permitted                                                                  
Nov 05 13:56:35  systemd[1]: Splunkd.service: Main process exited, code=exited, status=1/FAILURE    

The cited file /sys/fs/cgroup/cpu//system.slice/Splunkd.service does not exist ? Is this something the installer should have created ?

Tags (1)
1 Solution

yaunj
Engager

We had some issues running Splunk 8 as a non-privileged user (i.e. not root).

The directories /sys/fs/cgroup/cpu/system.slice/Splunkd.service and /sys/fs/cgroup/memory/system.slice/Splunkd.service are created when the Splunkd.service is started by systemd. They are used to influence the cgroup that the Splunk service is running under, and can be used to control CPU and Memory usage by the service.

In previous versions of Splunk, the systemd unit file generated by splunk enable boot-start -user splunk contained two ExecStartPost statements that changed the permissions for the cgroup directories. It also set User=splunk in the unit file. You can still see remains from this in the online documentation.

With Splunk 8, it seems that the Splunk process itself tries to manipulate these directories. When started with User=splunk (or a different non-privileged user), this is denied by the system, as these folders are owned by the root user. This also happens so soon in the startup process that Splunk terminates before the ExecStartPost hooks are run by systemd. If started without the User= statement in the unit file, Splunk starts as root, and drops privileges to SPLUNK_OS_USER from ${SPLUNK_HOME}/etc/splunk-launch.conf.

There are two options on how to fix this issue if you have been running Splunk as a non-privileged user via systemd:

  1. Don't use User=splunk in the unit file (and also drop the ExecStartPost statements). As long as SPLUNK_OS_USER=splunk in ${SPLUNK_HOME}/etc/splunk-launch.conf, Splunk will fix the permissions and drop privileges.
  2. Change the ExecStartPost statements to ExecStartPre. This ensures that systemd changes the permissions before Splunk is actually started.

View solution in original post

yaunj
Engager

We had some issues running Splunk 8 as a non-privileged user (i.e. not root).

The directories /sys/fs/cgroup/cpu/system.slice/Splunkd.service and /sys/fs/cgroup/memory/system.slice/Splunkd.service are created when the Splunkd.service is started by systemd. They are used to influence the cgroup that the Splunk service is running under, and can be used to control CPU and Memory usage by the service.

In previous versions of Splunk, the systemd unit file generated by splunk enable boot-start -user splunk contained two ExecStartPost statements that changed the permissions for the cgroup directories. It also set User=splunk in the unit file. You can still see remains from this in the online documentation.

With Splunk 8, it seems that the Splunk process itself tries to manipulate these directories. When started with User=splunk (or a different non-privileged user), this is denied by the system, as these folders are owned by the root user. This also happens so soon in the startup process that Splunk terminates before the ExecStartPost hooks are run by systemd. If started without the User= statement in the unit file, Splunk starts as root, and drops privileges to SPLUNK_OS_USER from ${SPLUNK_HOME}/etc/splunk-launch.conf.

There are two options on how to fix this issue if you have been running Splunk as a non-privileged user via systemd:

  1. Don't use User=splunk in the unit file (and also drop the ExecStartPost statements). As long as SPLUNK_OS_USER=splunk in ${SPLUNK_HOME}/etc/splunk-launch.conf, Splunk will fix the permissions and drop privileges.
  2. Change the ExecStartPost statements to ExecStartPre. This ensures that systemd changes the permissions before Splunk is actually started.

Spranta
Splunk Employee
Splunk Employee

Great Job!

Tags (1)
0 Karma

aoleske
Path Finder

Thanks! this solved my issue also...

0 Karma

nickhills
Ultra Champion

I wonder if running splunk disable boot-start followed by splunk enable boot-start -user splunk following the upgrade would repair the pre 8.x unit files?

If my comment helps, please give it a thumbs up!

MichaelPolla
Engager

Awesome, thanks. 😊

It worked. Note that in my case I also changed for SystemD :

splunk enable boot-start -systemd-managed 1 -user splunk

PS : I didn't find how to format the code like the others (inline, red text and pink background) 😪 that's the best I could do...

48tfhd86gv
Explorer

Thanks for the great insight @yaunj . I actually think your answer should be the accepted one as it provides much better explanation and background detail.

I have to say, that based on this and prior experiences, although Splunk is a great product, some aspects of their documentation and software development are of questionable quality.

0 Karma

DavidHourani
Super Champion

yeah totally agree with @48tfhd86gv, nice work @yaunj

0 Karma

sarweins
Explorer

I would also make sure everything on your instance is compatible with 8.0

0 Karma

DavidHourani
Super Champion

Hi @48tfhd86gv,

From "Error perform systemd preparation: Operation not permitted " Seems like the Splunk user is trying to create Splunkd.service and doesn't have the permissions. Try starting Splunk with the root user then reverting the file ownership back to the splunk user.

48tfhd86gv
Explorer

@DavidHourani Perfect, that worked. Thank you for saving me from the nightmare of having to re-install !

0 Karma

DavidHourani
Super Champion

You're welcome! Please accept the answer if it was helpful 🙂

0 Karma

ivanreis
Builder

Per the error message it seems to be a permission issue. The splunk process is trying to load, but the Debian is not allowing to start. Check which account was created to run the splunkd service. I believe you have the redeploy the ownership the user created to run splunkd service.

0 Karma

48tfhd86gv
Explorer

@Ivanreis, but the error says "Couldn't change ownership for /sys/fs/cgroup/cpu//system.slice/Splunkd.service", but "/sys/fs/cgroup/cpu//system.slice/Splunkd.service" does not even exist ???

0 Karma

ivanreis
Builder

I mean that it is possible the account that is being used to start the splunk service does not have all the required access to start splunkd service, so this process splunkd should run successfully, so if for some reason the user is not able to start this service, it is potential problem. Use the root user to try to redploy the permission for the user you have created to start splunk.
If it did not work, try to reinstall the previous version and after that redeploy Splunk 8 again.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...