We had some issues running Splunk 8 as a non-privileged user (i.e. not root).
The directories /sys/fs/cgroup/cpu/system.slice/Splunkd.service and /sys/fs/cgroup/memory/system.slice/Splunkd.service are created when the Splunkd.service is started by systemd. They are used to influence the cgroup that the Splunk service is running under, and can be used to control CPU and Memory usage by the service.
In previous versions of Splunk, the systemd unit file generated by splunk enable boot-start -user splunk contained two ExecStartPost statements that changed the permissions for the cgroup directories. It also set User=splunk in the unit file. You can still see remains from this in the online documentation.
With Splunk 8, it seems that the Splunk process itself tries to manipulate these directories. When started with User=splunk (or a different non-privileged user), this is denied by the system, as these folders are owned by the root user. This also happens so soon in the startup process that Splunk terminates before the ExecStartPost hooks are run by systemd. If started without the User= statement in the unit file, Splunk starts as root, and drops privileges to SPLUNK_OS_USER from ${SPLUNK_HOME}/etc/splunk-launch.conf .
There are two options on how to fix this issue if you have been running Splunk as a non-privileged user via systemd:
Don't use User=splunk in the unit file (and also drop the ExecStartPost statements). As long as SPLUNK_OS_USER=splunk in ${SPLUNK_HOME}/etc/splunk-launch.conf , Splunk will fix the permissions and drop privileges.
Change the ExecStartPost statements to ExecStartPre . This ensures that systemd changes the permissions before Splunk is actually started.
... View more