Installation

Why is Splunk not starting after upgrade to 8?

48tfhd86gv
Explorer

Went to upgrade splunk-7.3.0-657388c7a488-linux-2.6-amd64 to splunk-8.0.0-1357bef0a7f6-linux-2.6-amd64.deb on a standalone instance of Splunk running on Debian 10.1.

The install process went fine. The deb file recognised there was an existing vesion and shut it down before continuing. however now v8 just won't start.

Looking at journalctl -xe, I'm seeing the following :

Nov 05 13:56:35  splunk[597]: Couldn't change ownership for /sys/fs/cgroup/cpu//system.slice/Splunkd.service: Operation not permitted                     
Nov 05 13:56:35  splunk[597]: Error perform systemd preparation: Operation not permitted                                                                  
Nov 05 13:56:35  systemd[1]: Splunkd.service: Main process exited, code=exited, status=1/FAILURE    

The cited file /sys/fs/cgroup/cpu//system.slice/Splunkd.service does not exist ? Is this something the installer should have created ?

Tags (1)
1 Solution

yaunj
Engager

We had some issues running Splunk 8 as a non-privileged user (i.e. not root).

The directories /sys/fs/cgroup/cpu/system.slice/Splunkd.service and /sys/fs/cgroup/memory/system.slice/Splunkd.service are created when the Splunkd.service is started by systemd. They are used to influence the cgroup that the Splunk service is running under, and can be used to control CPU and Memory usage by the service.

In previous versions of Splunk, the systemd unit file generated by splunk enable boot-start -user splunk contained two ExecStartPost statements that changed the permissions for the cgroup directories. It also set User=splunk in the unit file. You can still see remains from this in the online documentation.

With Splunk 8, it seems that the Splunk process itself tries to manipulate these directories. When started with User=splunk (or a different non-privileged user), this is denied by the system, as these folders are owned by the root user. This also happens so soon in the startup process that Splunk terminates before the ExecStartPost hooks are run by systemd. If started without the User= statement in the unit file, Splunk starts as root, and drops privileges to SPLUNK_OS_USER from ${SPLUNK_HOME}/etc/splunk-launch.conf.

There are two options on how to fix this issue if you have been running Splunk as a non-privileged user via systemd:

  1. Don't use User=splunk in the unit file (and also drop the ExecStartPost statements). As long as SPLUNK_OS_USER=splunk in ${SPLUNK_HOME}/etc/splunk-launch.conf, Splunk will fix the permissions and drop privileges.
  2. Change the ExecStartPost statements to ExecStartPre. This ensures that systemd changes the permissions before Splunk is actually started.

View solution in original post

yaunj
Engager

We had some issues running Splunk 8 as a non-privileged user (i.e. not root).

The directories /sys/fs/cgroup/cpu/system.slice/Splunkd.service and /sys/fs/cgroup/memory/system.slice/Splunkd.service are created when the Splunkd.service is started by systemd. They are used to influence the cgroup that the Splunk service is running under, and can be used to control CPU and Memory usage by the service.

In previous versions of Splunk, the systemd unit file generated by splunk enable boot-start -user splunk contained two ExecStartPost statements that changed the permissions for the cgroup directories. It also set User=splunk in the unit file. You can still see remains from this in the online documentation.

With Splunk 8, it seems that the Splunk process itself tries to manipulate these directories. When started with User=splunk (or a different non-privileged user), this is denied by the system, as these folders are owned by the root user. This also happens so soon in the startup process that Splunk terminates before the ExecStartPost hooks are run by systemd. If started without the User= statement in the unit file, Splunk starts as root, and drops privileges to SPLUNK_OS_USER from ${SPLUNK_HOME}/etc/splunk-launch.conf.

There are two options on how to fix this issue if you have been running Splunk as a non-privileged user via systemd:

  1. Don't use User=splunk in the unit file (and also drop the ExecStartPost statements). As long as SPLUNK_OS_USER=splunk in ${SPLUNK_HOME}/etc/splunk-launch.conf, Splunk will fix the permissions and drop privileges.
  2. Change the ExecStartPost statements to ExecStartPre. This ensures that systemd changes the permissions before Splunk is actually started.

Spranta
Communicator

Great Job!

Tags (1)
0 Karma

aoleske
Path Finder

Thanks! this solved my issue also...

0 Karma

nickhills
Ultra Champion

I wonder if running splunk disable boot-start followed by splunk enable boot-start -user splunk following the upgrade would repair the pre 8.x unit files?

If my comment helps, please give it a thumbs up!

MichaelPolla
Engager

Awesome, thanks. 😊

It worked. Note that in my case I also changed for SystemD :

splunk enable boot-start -systemd-managed 1 -user splunk

PS : I didn't find how to format the code like the others (inline, red text and pink background) 😪 that's the best I could do...

0 Karma

48tfhd86gv
Explorer

Thanks for the great insight @yaunj . I actually think your answer should be the accepted one as it provides much better explanation and background detail.

I have to say, that based on this and prior experiences, although Splunk is a great product, some aspects of their documentation and software development are of questionable quality.

0 Karma

DavidHourani
Super Champion

yeah totally agree with @48tfhd86gv, nice work @yaunj

0 Karma

sarweins
Explorer

I would also make sure everything on your instance is compatible with 8.0

0 Karma

DavidHourani
Super Champion

Hi @48tfhd86gv,

From "Error perform systemd preparation: Operation not permitted " Seems like the Splunk user is trying to create Splunkd.service and doesn't have the permissions. Try starting Splunk with the root user then reverting the file ownership back to the splunk user.

48tfhd86gv
Explorer

@DavidHourani Perfect, that worked. Thank you for saving me from the nightmare of having to re-install !

0 Karma

DavidHourani
Super Champion

You're welcome! Please accept the answer if it was helpful 🙂

0 Karma

ivanreis
Builder

Per the error message it seems to be a permission issue. The splunk process is trying to load, but the Debian is not allowing to start. Check which account was created to run the splunkd service. I believe you have the redeploy the ownership the user created to run splunkd service.

0 Karma

48tfhd86gv
Explorer

@Ivanreis, but the error says "Couldn't change ownership for /sys/fs/cgroup/cpu//system.slice/Splunkd.service", but "/sys/fs/cgroup/cpu//system.slice/Splunkd.service" does not even exist ???

0 Karma

ivanreis
Builder

I mean that it is possible the account that is being used to start the splunk service does not have all the required access to start splunkd service, so this process splunkd should run successfully, so if for some reason the user is not able to start this service, it is potential problem. Use the root user to try to redploy the permission for the user you have created to start splunk.
If it did not work, try to reinstall the previous version and after that redeploy Splunk 8 again.

0 Karma
Get Updates on the Splunk Community!

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...