Stream is installed and the bash set_permissions.sh has been run as the root user. However, when splunkd is started, the streamfwd process also starts as the 'splunk' user and fails to capture.
Notes
/opt/splunkforwarder/etc/splunk-launch.conf specifies SPLUNK_OS_USER=splunk
If this is changed to SPLUNK_OS_USER=
then splunkd and streamfwd will start as root and stream WORKS
However, we do not want this, we want splunkd to run as the splunk user and only streamfwd to run as root user; this is supposed to happen once the set_permissions.sh script is run (yes we ran it as the root user also) but it does not work.
The mount point hosting the stream binary has a nosuid bit set. This setting both ignores setuid attributes that are set on files and also prevents capabilities assigned by 'setcap' from being honoured. Running the following command finds the mount point relevant for the streamfwd binaries location and also the settings for that mount point in fstab:
[/opt/splunkforwarder/etc/apps/Splunk_TA_stream/linux_x86_64/bin]$ findmnt -T .
TARGET SOURCE FSTYPE OPTIONS
/opt /dev/mapper/data-opt xfs rw,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota
Removing this 'nosuid' flag from the mount point in fstab and remounting will allow the setcap OR setuid operations in the bash script to work correctly.
The mount point hosting the stream binary has a nosuid bit set. This setting both ignores setuid attributes that are set on files and also prevents capabilities assigned by 'setcap' from being honoured. Running the following command finds the mount point relevant for the streamfwd binaries location and also the settings for that mount point in fstab:
[/opt/splunkforwarder/etc/apps/Splunk_TA_stream/linux_x86_64/bin]$ findmnt -T .
TARGET SOURCE FSTYPE OPTIONS
/opt /dev/mapper/data-opt xfs rw,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota
Removing this 'nosuid' flag from the mount point in fstab and remounting will allow the setcap OR setuid operations in the bash script to work correctly.