cgroup error when Splunk Universal Forwarder v9.4.4 is installed

[ArtieZ]

Hello,

I wanted to test UF v9.4.4 and installed it on RHEL8 and RHEL9 instances.  UF v9.4.4 seems to work in our environment and it sends logs which are searchable on Search Head.

However, I can see errors in splunkd.log happening at start up. The errors are logged on start up only.

RHEL9
ERROR SystemInfo [1021 MainThread] - Failed to read memory limit at location="V2:/sys/fs/cgroup:/system.slice/SplunkForwarder.service:/sys/fs/cgroup:/system.slice/SplunkForwarder.service:" 

RHEL8
ERROR SystemInfo [1835 MainThread] - Failed to read memory limit at location="V1:/sys/fs/cgroup/cpu,cpuacct:/system.slice/splunk.service:/sys/fs/cgroup/memory:/system.slice/splunk.s
ervice:" 

I have done a little bit of troubleshooting:

1. tried setting selinux to 0 - no change

2. checked the permissions - splunk user has access to dirs/files mentioned in the errors

3. Checked whether the files /sys/fs/cgroup/system.slice/splunk.service/memory.max (for v2) and  /sys/fs/cgroup/memory/memory.limit_in_bytes (for v1) exist - they do exist

4. Both files have "max" in them. 

I suspect this could be a bug.  Please let me know if any ideas on these errors.

EDIT: also tested with UF v9.4.5 and v10.0.0 - same errors in the logs at start up

[vjdev]

Hello,

Try restart, if you have not done [SELINUX settings to apply]>

[/usr/lib/systemd/system/SplunkForwarder.service] OR [/etc/systemd/system/SplunkForwarder.service]

[Service]
MemoryLimit=16542720000 [16 GB In Byte | modify as per your system memory ]

sudo systemctl daemon-reload


sudo systemctl restart SplunkForwarder

Thank You!

[ArtieZ]

Thanks for your reply vjdev.

I have tried SELinux in permanent and rebooting - no difference.

Setting MemroyLimit in the service file did not make any difference. Also, this option would be problematic in my scenario because of the large number of instances with different specs and different applications/services running on them, so calculating in advance and automating the deployment of the value would be challenging.