Hi all, when upgrading to Splunk Enterprise 9.4.15, 10.0.10, 10.2.7 or 10.4.3 with the .rpm or .deb package, the upgrade may fail on hardened systems with the following error: runuser: failed to execute /var/tmp/splunk-preinstall.EDCpCC/temp_splunk-preinstall: Permission denied Root cause The preinstall script in these packages was changed to address an internal Splunk security issue tracked as VULN-81607. Before the change, the script extracted the embedded splunk-preinstall binary into $SPLUNK_HOME/bin and ran it as root. Now it extracts the binary into a temporary directory under /var/tmp (falling back to /tmp) and runs it either as root or as the owner of the Splunk installation, depending on the ownership of the install tree. On CIS-hardened systems, /var/tmp and /tmp are mounted with the noexec option. The kernel refuses to execute any binary from these file systems, so the preinstall check fails and the package manager aborts the whole upgrade. The script only handles the /tmp case in its comment and does not consider that /var/tmp is commonly hardened the same way. The relevant CIS recommendations are, for example in the CIS Red Hat Enterprise Linux 8 Benchmark v2.0: - Linux 1.1.4.1: Ensure that a separate partition exists for /var/tmp - Linux 1.1.4.2: Ensure that the noexec option is set on the /var/tmp partition Reference: https://www.ibm.com/docs/en/powersc-standard/2.2.0?topic=scac-cis-red-hat-enterprise-linux-8-benchmark-v20 Here is the relevant snippet from the preinstall script: # VULN-81607: Root is trusted, but a malicious service account may control existing install-tree paths.
# Only run existing install content as root when the full launcher ancestry is root-owned and not group/world-writable.
# splunk-preinstall is package content, but it calls the existing launcher during KVStore checks.
if [ -x "$SPLUNK_HOME/bin/splunk" ] ; then
echo "This looks like an upgrade of an existing Splunk Server. Checking to see what component we are installing"
if [ "$COMPONENT" = "splunk" ]; then
echo "extracting splunk_preinstall_base64 into a temporary directory"
# Prefer /var/tmp because some hardened hosts mount /tmp noexec.
splunk_execution_owner="$(resolve_splunk_execution_owner || true)"
preinstall_tmpdir=""
for preinstall_tmpbase in /var/tmp /tmp; do
preinstall_tmpdir="$(mktemp -d "$preinstall_tmpbase/splunk-preinstall.XXXXXX" 2>/dev/null)" && break
done Who is affected Only upgrades of an existing installation are affected, because the check runs only when $SPLUNK_HOME/bin/splunk already exists. Fresh installations are not affected. The paths are hard-coded in the script, so setting TMPDIR does not help. Workaround Temporarily remount /var/tmp with exec for the duration of the upgrade, then restore the hardened mount options: mount -o remount,exec /var/tmp
rpm -Uvh splunk-10.4.3-4174a2deda5d.x86_64.rpm # or dpkg -i for .deb
mount -o remount,noexec /var/tmp
... View more