systemctl stop Splunkd hangs

[harryvdtol]

Hello,

Since a few months we are facing an issue with stopping Splunk on Red Hat Linux-rel8.

We do "systemctl stop Splunkd" to stop the Splunk proces.
In most cases Splunks stops and the systemctl prompts comes back.

But sometimes (let say 1 out of 10) Splunk stops, but the systemctl prompt does not comes back.

Then, after 6 minues (the timeout in the Splunkd.service) systemctl comes back
In /var/log/messages i see this after 6 minutes.

Splunkd.service: Failed with result 'timeout'.
Stopped Systemd service file for Splunk, generated by 'splunk enable boot-start'.

In the splunkd.log i can see that Splunk has stopped. No Splunk proces is running.
With "ps -ef | grep splunk" i can see that there a no Splunk processes running.
"ps -ef | grep systemctl" i can see that systemctl is still running.

It happens on Search cluster, index cluster, Heavy Forwarders etc.

Splunk support says is it an Red Hat Linux issue and Red Hat points to Splunk.

I wonder if we are the only one who is having this issue.

Any remarks are appreciated.

Regards,

Harry

[harryvdtol]

Hello,

Hereby a new update on this case
Some weeks ago we have upgraded Splunk to 941.
After the upgrade we receive erros when executing Splunk commands like splunk show .. or btool

Failed to calculate cpu count from cgroup location="V2:/sys/fs/cgroup:/user.slice/user-570057916.slice/session-9.scope:/sys/fs/cgroup:/user.slice/user-570057916.slice/session-9.scope:"

Because the Splunk version 941 added support for cgroups V2, we removed the workaround.
Since then the original issue is back: adhoc hanging of systemctl

If anyone has the issue, i really would like to know.

Regards,

Harry