Splunk, Ubuntu 21.10, and the switch to cgroups V2

[da_667]

Hey everyone,

Long time user, first time poster. So here's the root of my problem:

I'm supporting users in a practice environment who wanted to use Ubuntu 21.10 server to host Splunk. The install goes great right up until the point in which users are instructed to run

/opt/splunk/bin/splunk enable boot-start -systemd-managed 1

The splunk binary creates the Splunkd.service file as you'd expect. But it doesn't actually work:

root@short-term-test:/home/ayy# systemctl start Splunkd.service
Job for Splunkd.service failed because the control process exited with error code.
See "systemctl status Splunkd.service" and "journalctl -xeu Splunkd.service" for details.

and of course, the journal doesn't actually tell you anything other than the service failed. But /var/log/syslog gives you enough breadcrumbs to figure out the root the problem:

Jan 11 20:06:18 short-term-test bash[7690]: chown: cannot access '/sys/fs/cgroup/cpu/system.slice/Splunkd.service': No such file or directory

so we go to /etc/systemd/system/Splunkd.service:

root@short-term-test:/home/ayy# cat /etc/systemd/system/Splunkd.service
#This unit file replaces the traditional start-up script for systemd
#configurations, and is used when enabling boot-start for Splunk on
#systemd-based Linux distributions.

[Unit]
Description=Systemd service file for Splunk, generated by 'splunk enable boot-start'
After=network.target

[Service]
Type=simple
Restart=always
ExecStart=/opt/splunk/bin/splunk _internal_launch_under_systemd
KillMode=mixed
KillSignal=SIGINT
TimeoutStopSec=360
LimitNOFILE=65536
SuccessExitStatus=51 52
RestartPreventExitStatus=51
RestartForceExitStatus=52
User=splunk
Group=splunk
Delegate=true
CPUShares=1024
MemoryLimit=2073997312
PermissionsStartOnly=true
ExecStartPost=/bin/bash -c "chown -R splunk:splunk /sys/fs/cgroup/cpu/system.slice/%n"
ExecStartPost=/bin/bash -c "chown -R splunk:splunk /sys/fs/cgroup/memory/system.slice/%n"

[Install]
WantedBy=multi-user.target

And we see in the ExecStartPost, attempts to change file ownership for all the cgroup files under /sys/fs/cgroup/cpu/system.slice/Splunkd.service, and /sys/fs/cgroup/memory/system.slice/Splunkd.service, where /var/log/syslog says the service is failing.

So, I searched through the community posts and discovered this is something of a known problem:

https://community.splunk.com/t5/Installation/Systemd-broken-on-new-install/m-p/482885#M10392

https://community.splunk.com/t5/Installation/Why-is-Splunk-not-starting-after-upgrade-to-8/m-p/47882...

From these two posts, there appear to be two schools of thought:

-splunk hardcodes the cgroup directories it thinks the service should be using. So if they're wrong, find them yourself.

From my brief experiences with Ubuntu 21.10 and the new cgroup setup, it appears that the files are located under /sys/fs/cgroup/system.slice/Splunkd.service/

-splunk doesn't need the chown command if /opt/splunk/etc/splunk-launch.conf is set to lower its privileges to the splunk user. (SPLUNK_OS_USER=splunk)

If I comment out the ExecStartPost statements from Splunkd.service, and use systemctl daemon-reload to get the system to honor the service file changes, Splunk starts normally, sort of supporting this idea that splunk can handle its cgroup file permissions internally.

This brings me to my actual questions:

Would there be any ill-effects of changing the Splunkd.service script to change file ownership of all the files in /sys/fs/cgroup/system.slice/Splunkd.service/ directory to the splunk user? or are there specific files and parameters that I need to leave alone?

Can I instead remove/disable the ExecStartPost statements and rely on the SPLUNK_OS_USER parameter in /opt/splunk/etc/splunk-launch.conf instead?

Is there some sort of a plan to fix the systemd Splunkd.service file? I understand that Ubuntu 21.10 is NOT a long-term service release BUT I have a feeling that this adoption of cgroups v2, and the new cgroups layout is going to be prevalent change in the next long-term service release (e.g. 22.04 and beyond), and I really don't want to have to do this work-around for every fresh Splunk install in the future.

Thank you for your time and consideration.

[gavins_k1]

Hi da_667 , 

I just had the same problem on 22.04, and the 'comment out the ExecStartPost' trick you mentioned worked for me, no I'll effects so far. Thanks for your help 😁👍