Splunk, Ubuntu 21.10, and the switch to cgroups V2

New Member

Hey everyone,


Long time user, first time poster. So here's the root of my problem:

I'm supporting users in a practice environment who wanted to use Ubuntu 21.10 server to host Splunk. The install goes great right up until the point in which users are instructed to run

/opt/splunk/bin/splunk enable boot-start -systemd-managed 1

The splunk binary creates the Splunkd.service file as you'd expect. But it doesn't actually work:



root@short-term-test:/home/ayy# systemctl start Splunkd.service
Job for Splunkd.service failed because the control process exited with error code.
See "systemctl status Splunkd.service" and "journalctl -xeu Splunkd.service" for details.




and of course, the journal doesn't actually tell you anything other than the service failed. But /var/log/syslog gives you enough breadcrumbs to figure out the root the problem:



Jan 11 20:06:18 short-term-test bash[7690]: chown: cannot access '/sys/fs/cgroup/cpu/system.slice/Splunkd.service': No such file or directory



so we go to /etc/systemd/system/Splunkd.service:



root@short-term-test:/home/ayy# cat /etc/systemd/system/Splunkd.service
#This unit file replaces the traditional start-up script for systemd
#configurations, and is used when enabling boot-start for Splunk on
#systemd-based Linux distributions.

Description=Systemd service file for Splunk, generated by 'splunk enable boot-start'

ExecStart=/opt/splunk/bin/splunk _internal_launch_under_systemd
SuccessExitStatus=51 52
ExecStartPost=/bin/bash -c "chown -R splunk:splunk /sys/fs/cgroup/cpu/system.slice/%n"
ExecStartPost=/bin/bash -c "chown -R splunk:splunk /sys/fs/cgroup/memory/system.slice/%n"





And we see in the ExecStartPost, attempts to change file ownership for all the cgroup files under /sys/fs/cgroup/cpu/system.slice/Splunkd.service, and /sys/fs/cgroup/memory/system.slice/Splunkd.service, where /var/log/syslog says the service is failing.

So, I searched through the community posts and discovered this is something of a known problem:


From these two posts, there appear to be two schools of thought:

-splunk hardcodes the cgroup directories it thinks the service should be using. So if they're wrong, find them yourself.

From my brief experiences with Ubuntu 21.10 and the new cgroup setup, it appears that the files are located under /sys/fs/cgroup/system.slice/Splunkd.service/

-splunk doesn't need the chown command if /opt/splunk/etc/splunk-launch.conf is set to lower its privileges to the splunk user. (SPLUNK_OS_USER=splunk)

If I comment out the ExecStartPost statements from Splunkd.service, and use systemctl daemon-reload to get the system to honor the service file changes, Splunk starts normally, sort of supporting this idea that splunk can handle its cgroup file permissions internally.


This brings me to my actual questions:

Would there be any ill-effects of changing the Splunkd.service script to change file ownership of all the files in /sys/fs/cgroup/system.slice/Splunkd.service/ directory to the splunk user? or are there specific files and parameters that I need to leave alone?

Can I instead remove/disable the ExecStartPost statements and rely on the SPLUNK_OS_USER parameter in /opt/splunk/etc/splunk-launch.conf instead?

Is there some sort of a plan to fix the systemd Splunkd.service file? I understand that Ubuntu 21.10 is NOT a long-term service release BUT I have a feeling that this adoption of cgroups v2, and the new cgroups layout is going to be prevalent change in the next long-term service release (e.g. 22.04 and beyond), and I really don't want to have to do this work-around for every fresh Splunk install in the future.


Thank you for your time and consideration.

Labels (2)
0 Karma

Path Finder

Hi da_667 , 

I just had the same problem on 22.04, and the 'comment out the ExecStartPost' trick you mentioned worked for me, no I'll effects so far. Thanks for your help 😁👍

0 Karma
Get Updates on the Splunk Community!

SplunkTrust | Where Are They Now - Michael Uschmann

The Background Five years ago, Splunk published several videos showcasing members of the SplunkTrust to share ...

Admin Your Splunk Cloud, Your Way

Join us to maximize different techniques to best tune Splunk Cloud. In this Tech Enablement, you will get ...

Cloud Platform | Discontinuing support for TLS version 1.0 and 1.1

Overview Transport Layer Security (TLS) is a security communications protocol that lets two computers, ...