The splunk binary creates the Splunkd.service file as you'd expect. But it doesn't actually work:
root@short-term-test:/home/ayy# systemctl start Splunkd.service
Job for Splunkd.service failed because the control process exited with error code.
See "systemctl status Splunkd.service" and "journalctl -xeu Splunkd.service" for details.
and of course, the journal doesn't actually tell you anything other than the service failed. But /var/log/syslog gives you enough breadcrumbs to figure out the root the problem:
Jan 11 20:06:18 short-term-test bash: chown: cannot access '/sys/fs/cgroup/cpu/system.slice/Splunkd.service': No such file or directory
so we go to /etc/systemd/system/Splunkd.service:
root@short-term-test:/home/ayy# cat /etc/systemd/system/Splunkd.service
#This unit file replaces the traditional start-up script for systemd
#configurations, and is used when enabling boot-start for Splunk on
#systemd-based Linux distributions.
Description=Systemd service file for Splunk, generated by 'splunk enable boot-start'
ExecStartPost=/bin/bash -c "chown -R splunk:splunk /sys/fs/cgroup/cpu/system.slice/%n"
ExecStartPost=/bin/bash -c "chown -R splunk:splunk /sys/fs/cgroup/memory/system.slice/%n"
And we see in the ExecStartPost, attempts to change file ownership for all the cgroup files under /sys/fs/cgroup/cpu/system.slice/Splunkd.service, and /sys/fs/cgroup/memory/system.slice/Splunkd.service, where /var/log/syslog says the service is failing.
So, I searched through the community posts and discovered this is something of a known problem:
From these two posts, there appear to be two schools of thought:
-splunk hardcodes the cgroup directories it thinks the service should be using. So if they're wrong, find them yourself.
From my brief experiences with Ubuntu 21.10 and the new cgroup setup, it appears that the files are located under /sys/fs/cgroup/system.slice/Splunkd.service/
-splunk doesn't need the chown command if /opt/splunk/etc/splunk-launch.conf is set to lower its privileges to the splunk user. (SPLUNK_OS_USER=splunk)
If I comment out the ExecStartPost statements from Splunkd.service, and use systemctl daemon-reload to get the system to honor the service file changes, Splunk starts normally, sort of supporting this idea that splunk can handle its cgroup file permissions internally.
This brings me to my actual questions:
Would there be any ill-effects of changing the Splunkd.service script to change file ownership of all the files in /sys/fs/cgroup/system.slice/Splunkd.service/ directory to the splunk user? or are there specific files and parameters that I need to leave alone?
Can I instead remove/disable the ExecStartPost statements and rely on the SPLUNK_OS_USER parameter in /opt/splunk/etc/splunk-launch.conf instead?
Is there some sort of a plan to fix the systemd Splunkd.service file? I understand that Ubuntu 21.10 is NOT a long-term service release BUT I have a feeling that this adoption of cgroups v2, and the new cgroups layout is going to be prevalent change in the next long-term service release (e.g. 22.04 and beyond), and I really don't want to have to do this work-around for every fresh Splunk install in the future.