Summary: On a CentOS Stream 9 system, after installing Splunk in /opt/splunk and configuring it to start on boot with systemd, I've noticed unusual behavior. Using manual Splunk commands (/opt/splun...
See more...
Summary: On a CentOS Stream 9 system, after installing Splunk in /opt/splunk and configuring it to start on boot with systemd, I've noticed unusual behavior. Using manual Splunk commands (/opt/splunk/bin/splunk [start | stop | restart]) alters the Splunkd.service file in /etc/systemd/system/, creating a timestamped backup. This change prevents Splunk from starting using systemctl commands and consequently on boot, defeating the purpose of the systemd setup. Using chattr to make the service file immutable is a current workaround. This behavior seems specific to CentOS Stream 9. How to recreate issue: On a centos stream 9 machine, installed splunk under /opt/splunk, and run splunk as user 'splunk'. Enable boot-start with systemd-managed 1, after stopping Splunk. After enabling boot-start, a file will be created at /etc/systemd/system/Splunkd.service. Starting and stopping splunk using systemctl works fine, and normal. However, if you run sudo /opt/splunk/bin/splunk [start | stop | restart], splunk itself will change the/etc/systemd/system/Splunkd.service, and create a backup with a timestamp, e.g. Splunkd.service_2023_09_21_06_49_05. When trying to start with systemctl again: e.g. sudo systemctl start Splunkd Failed to start Splunkd.service: Unit Splunkd.service failed to load properly, please adjust/correct and reload service manager: Device or resource busy See system logs and 'systemctl status Splunkd.service' for details. This will lead to Splunk not starting after reboot, which is the whole point of enabling systemd. This error message shows up, because the Splunkd.service file has been altered. To get systemctl working again, i run sudo systemctl daemon-reload But as soon as one tries to do a manual start|stop|restart command, the same issue arises. When diffing the new service file and old service file: diff Splunkd.service Splunkd.service_2023_09_21_06_49_05 26c26 < MemoryLimit=3723374592 --- > MemoryLimit=3723378688 memoryLimit is the only value that is changed for each subsequent 'backup' of the service file. It just switches between these two values Mr chat.gpt suggested to make the service file non-immutable with sudo chattr +i /etc/systemd/system/Splunkd.service After this change, whenever doing manual start | stop | restart, you get a WARNING message: But it won't **bleep** up your Service file, and hence splunk will start after reboot. So it is Splunk itself who is changing the Service file. However, this issue was discovered in Centos Stream 9, and cannot be replicated in earlier versions. Anybody know what may have caused this weird error?