Security issue?! Splunk UF v9.x is re-adding readall capability

[schose]

Hi all,

Splunk UF since 9.x is setting 

[Service]
NoNewPrivileges=yes
AmbientCapabilities=CAP_DAC_READ_SEARCH

in systemd unit file (/etc/systemd/system/SplunkForwarder.service). This enables splunkforwarder to bypass Filesystems permissions and acls and read every file on harddisk - yes, every file: every ssh key, every private key, confidential data.. the opposite of the "least-to-know" principle. 

As we have correct filesystem permissions in place we decided to remove those settings from systemd unit file. When we now run e.g.: "/opt/splunkforwarder/bin/splunk stop" command the systemd file is rewritten by the splunk command. This will start splunkforwarder with enabled CAP_DAC_READ_SEARCH capability.

To make is more visual we uploaded a video to https://asciinema.org/a/FAYFPJYrKaizfL3alzvm3uNGF . 

Are you able to reproduce the issue? What do you think?

For us this looks like a secuity issue, as we would never expect a command like "splunk stop" manipulate systemd files. I'm also not aware which other command might rewrite the systemd unit. I also do not seed any usecase for this. 

steps to reproduce:
install-splunkuf.sh

#!/bin/bash

# break if errors
set -e

# add system user
sudo groupadd splunk
sudo useradd splunk --system --home-dir /opt/splunk --create-home -g splunk

wget -O /tmp/splunkuf.tgz https://download.splunk.com/products/universalforwarder/releases/9.1.0/linux/splunkforwarder-9.1.0-1c86ca0bacc3-Linux-x86_64.tgz
#wget -O /tmp/splunkuf.tgz https://download.splunk.com/products/universalforwarder/releases/9.0.5/linux/splunkforwarder-9.0.5-e9494146ae5c-Linux-armv8.tgz

tar zxfv /tmp/splunkuf.tgz -C /opt
echo -e "[user_info]\nUSERNAME=admin\nPASSWORD=Password01" > /opt/splunkforwarder/etc/system/local/user-seed.conf
/opt/splunkforwarder/bin/splunk start --accept-license && /opt/splunkforwarder/bin/splunk stop -f
/opt/splunkforwarder/bin/splunk enable boot-start -user splunk -group splunk -systemd-managed 1

# remove capabilities from systemd service
sed -i '/^NoNewPrivileges\|^AmbientCapabilities/s/^/#/' /etc/systemd/system/SplunkForwarder.service
systemctl daemon-reload

systemctl start SplunkForwarder
systemctl status SplunkForwarder

# systemd file is still fine
echo -n "systemd unit file after starting splunk"
cat /etc/systemd/system/SplunkForwarder.service

pid=$(systemctl show -p MainPID --value SplunkForwarder.service) && getpcaps $pid

when you now run 

/opt/splunkforwarder/bin/splunk stop
cat /etc/systemd/system/SplunkForwarder.service

you see that lines

NoNewPrivileges=yes
AmbientCapabilities=CAP_DAC_READ_SEARCH

are re-added to /etc/systemd/system/SplunkForwarder.service and next time the service is started caps are set. A backup file is also placed /etc/systemd/system/SplunkForwarder.service_TIMESTAMP.

when running a strace

strace -s 0 -o /tmp/910stop.strace -f /opt/splunkforwarder/bin/splunk stop

we clearly see the splunk process manipulating the systemd file.

2120 rename("/etc/systemd/system/SplunkForwarder.service", "/etc/systemd/system/SplunkForwarder.service_2023_07_03_21_47_00") = 0
2120 clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7feb05354f10) = 2122
2120 wait4(2122, 
2122 set_robust_list(0x7feb05354f20, 24) = 0

This happens on all 9.x versions of UF. 

best regards,

Andreas

[inventsekar]

well, good catch @schose ! 

nice learning, sir! 

[schose · Accepted solution]

Hi all,

To ensure that CAP_DAC_READ_SEARCH is not set, systemd overwrite mechanism could be used.

create /etc/systemd/system/SplunkForwarder.service.d/override.conf with content:

[Service]
NoNewPrivileges=yes
AmbientCapabilities=

this ensures, that even if /etc/systemd/system/SplunkForwarder.service is rewritten - which still looks like an issue to me - the AmbientCapabilities are still empty. 

Best regards,

Andreas