Installation

Does Splunk support cgroups v2 for Workload Management?

AlexHaydock
Engager

I'm looking to enable Workload Management for Splunk and I'm just trying to understand whether this is fully supported on a deployment which is using cgroups v2.

Based on the documentation, the wording on the Configure Linux systemd for workload management page suggests that Splunk understands cgroups v1:

CPU: /sys/fs/cgroup/cpu/system.slice/<SPLUNK_SERVER_NAME>.service
Memory: /sys/fs/cgroup/memory/system.slice/<SPLUNK_SERVER_NAME>.service

But there is nothing (except the diagram) that suggests it can operate under cgroups v2 as well.

Does anyone know whether Splunk fully supports cgroups v2, or will our deployment need to be cgroups v1?

Edit: Some investigation suggests that Splunk fails to start with Workload Management enabled on a cgroups v2 host:

Jul 15 10:30:50 hostnamehere splunk[16311]: Couldn't open dir /sys/fs/cgroup/cpu/: No such file or directory
Jul 15 10:30:50 hostnamehere splunk[16311]: Error perform systemd preparation: No such file or directory
Labels (1)

nortonjco
Explorer

This is the same thing I needed to do for RHEL 9.

0 Karma

cameronjust
Path Finder

I just encountered this issue with Fedora 33

My fix was to edit /etc/systemd/system/Splunkd.service then change the ExecStartPost commands to 

 

 

# old cgroups v1 method installed by Splunk. Comment these out
# ExecStartPost=/bin/bash -c "chown -R splunk:splunk /sys/fs/cgroup/cpu/system.slice/%n"
# ExecStartPost=/bin/bash -c "chown -R splunk:splunk /sys/fs/cgroup/memory/system.slice/%n"

# new cgroups v2 method
ExecStartPost=/bin/bash -c "chown -R splunk:splunk /sys/fs/cgroup/system.slice/%n"

 


Don't forget to run "systemctl daemon-reload" after changing the file.

Edit: Not sure if there are other implications with cgroups v2 as I haven't tested workload management but at least Splunk starts with that fix.

 

 

Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Laser Bananas and Edge Hubs: Exploring Operational Technology (OT) Data Through a ...

  OT is a different environment to traditional IT and can have interesting challenges when interfacing the ...

Event Series: Mastering AI Tokenomics and Splunk Agent Observability

Beyond the Black Box: Correlating AI Performance and Tokenomics with Splunk Agent Observability   As ...

span_metrics: The OpenTelemetry-Idiomatic Way to See Inside Your Services

You open a trace in Splunk Observability Cloud and everything looks fine. One root span, order-pipeline, with ...