Installation

Security issue?! Splunk UF v9.x is re-adding readall capability

schose
Builder

Hi all,

Splunk UF since 9.x is setting 

[Service]
NoNewPrivileges=yes
AmbientCapabilities=CAP_DAC_READ_SEARCH

in systemd unit file (/etc/systemd/system/SplunkForwarder.service). This enables splunkforwarder to bypass Filesystems permissions and acls and read every file on harddisk - yes, every file: every ssh key, every private key, confidential data.. the opposite of the "least-to-know" principle. 

As we have correct filesystem permissions in place we decided to remove those settings from systemd unit file. When we now run e.g.: "/opt/splunkforwarder/bin/splunk stop" command the systemd file is rewritten by the splunk command. This will start splunkforwarder with enabled CAP_DAC_READ_SEARCH capability.

To make is more visual we uploaded a video to https://asciinema.org/a/FAYFPJYrKaizfL3alzvm3uNGF . 

Are you able to reproduce the issue? What do you think?

For us this looks like a secuity issue, as we would never expect a command like "splunk stop" manipulate systemd files. I'm also not aware which other command might rewrite the systemd unit. I also do not seed any usecase for this. 

steps to reproduce:
install-splunkuf.sh

#!/bin/bash

# break if errors
set -e

# add system user
sudo groupadd splunk
sudo useradd splunk --system --home-dir /opt/splunk --create-home -g splunk

wget -O /tmp/splunkuf.tgz https://download.splunk.com/products/universalforwarder/releases/9.1.0/linux/splunkforwarder-9.1.0-1c86ca0bacc3-Linux-x86_64.tgz
#wget -O /tmp/splunkuf.tgz https://download.splunk.com/products/universalforwarder/releases/9.0.5/linux/splunkforwarder-9.0.5-e9494146ae5c-Linux-armv8.tgz

tar zxfv /tmp/splunkuf.tgz -C /opt
echo -e "[user_info]\nUSERNAME=admin\nPASSWORD=Password01" > /opt/splunkforwarder/etc/system/local/user-seed.conf
/opt/splunkforwarder/bin/splunk start --accept-license && /opt/splunkforwarder/bin/splunk stop -f
/opt/splunkforwarder/bin/splunk enable boot-start -user splunk -group splunk -systemd-managed 1

# remove capabilities from systemd service
sed -i '/^NoNewPrivileges\|^AmbientCapabilities/s/^/#/' /etc/systemd/system/SplunkForwarder.service
systemctl daemon-reload

systemctl start SplunkForwarder
systemctl status SplunkForwarder

# systemd file is still fine
echo -n "systemd unit file after starting splunk"
cat /etc/systemd/system/SplunkForwarder.service

pid=$(systemctl show -p MainPID --value SplunkForwarder.service) && getpcaps $pid

 

when you now run 

 

/opt/splunkforwarder/bin/splunk stop
cat /etc/systemd/system/SplunkForwarder.service

you see that lines

NoNewPrivileges=yes
AmbientCapabilities=CAP_DAC_READ_SEARCH

are re-added to /etc/systemd/system/SplunkForwarder.service and next time the service is started caps are set. A backup file is also placed /etc/systemd/system/SplunkForwarder.service_TIMESTAMP.

when running a strace

strace -s 0 -o /tmp/910stop.strace -f /opt/splunkforwarder/bin/splunk stop

we clearly see the splunk process manipulating the systemd file.

2120 rename("/etc/systemd/system/SplunkForwarder.service", "/etc/systemd/system/SplunkForwarder.service_2023_07_03_21_47_00") = 0
2120 clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7feb05354f10) = 2122
2120 wait4(2122, 
2122 set_robust_list(0x7feb05354f20, 24) = 0

 

This happens on all 9.x versions of UF. 

best regards,

Andreas

Labels (2)
1 Solution

schose
Builder

Hi all,

To ensure that CAP_DAC_READ_SEARCH is not set, systemd overwrite mechanism could be used.

create /etc/systemd/system/SplunkForwarder.service.d/override.conf with content:

[Service]
NoNewPrivileges=yes
AmbientCapabilities=

 

this ensures, that even if /etc/systemd/system/SplunkForwarder.service is rewritten - which still looks like an issue to me - the AmbientCapabilities are still empty. 

Best regards,

Andreas

View solution in original post

0 Karma

inventsekar
SplunkTrust
SplunkTrust

well, good catch @schose ! 

nice learning, sir! 

0 Karma

schose
Builder

Hi all,

To ensure that CAP_DAC_READ_SEARCH is not set, systemd overwrite mechanism could be used.

create /etc/systemd/system/SplunkForwarder.service.d/override.conf with content:

[Service]
NoNewPrivileges=yes
AmbientCapabilities=

 

this ensures, that even if /etc/systemd/system/SplunkForwarder.service is rewritten - which still looks like an issue to me - the AmbientCapabilities are still empty. 

Best regards,

Andreas

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...