Splunk App for Stream: Why does set_permissions.sh selectively set the permissions on one file when inputs.conf uses a different file?

[gjanders]

On a Linux server we have, we used the set_permissions.sh script packaged with the Splunk App for Stream.

This used the logic in the script chown to root and chmod the file streamfwd-rhel6.

Unfortunately, the streamfwd was the binary running, so I had to manually have the server administrator run:

chown root /opt/splunkforwarder/etc/apps/Splunk_TA_stream/linux_x86_64/bin/streamfwd
chmod 4711 /opt/splunkforwarder/etc/apps/Splunk_TA_stream/linux_x86_64/bin/streamfwd

To resolve my issue, the inputs.conf just has the default:

[streamfwd]
source = stream

With an override for the disabled = 0 so that it actually runs.

Why does the script choose to change permissions on 1 file, but the script that runs the input does not choose the same file? It looks like a bug, so I'm wondering if anyone else has this issue...

The application itself appears to be working fine after a forwarder restart and fixing the permissions...

[gjanders · Accepted solution]

I'm going to answer my own question here, the Splunk_TA_stream/linux_x86_64/bin/streamfwd-rhel6 is only used when dedicated capture mode is enabled, by default it is not used.

Therefore the bug is in the Splunk_TA_stream/set_permissions.sh script, it would appear to only set the run as root privileges on the streamfwd-rhel6 binary in some circumstances and this is the cause of my issue.

It's easy enough to work around so I will not log a Splunk support case on this, but I hope they will modify the script at some point to work as expected, I tested on Splunk Stream 7.0

[vshcherbakov_sp]

Not sure I understand what the issue is. You need to restart Splunk or restart streamfwd modular input for the permission changes made by set_permission.sh script to take effect.

[gjanders]

That is correct, but the set_permissions.sh script which Splunk provides sets permissions on the splunkfwd-rhel6 file no amount of restarting will fix that.

I had to manually change permissions on streamfwd

Refer to:
Splunk app for stream installation instructions

"Ensure Splunk_TA_stream privileges"
"On *nix, if you prefer that splunkd not run as root, you can use the set_permissions.sh script to set the required OS-level privileges to streamfwd only: "

The above does not work as expected, either the streamfwd-rhel6 is the one that should be executed automatically by the Splunk_TA_stream or the set_permissions.sh is changing permissions on the wrong binary file!

[vshcherbakov_sp]

streamfwd is just a symlink that points to either streamfwd-rhel5 or streamfwd-rhel6 depending on the capture mode (agent/dedicated).

On modern Linux versions, set_permissions.sh scripts configures streamfwd-rhel5 binary with Linux capabilities needed to run promiscuous packet capture, since it's more secure compared to setuid as root.

streamfwd-rhel6 is a different case altogether, and it does need to be run as root.

On older Linux machines, the script sets both rhel5 and rhel6 binaries to run as root.

[gjanders]

In regard to your above response "On modern Linux versions, set_permissions.sh scripts configures streamfwd-rhel5 binary with Linux capabilities needed to run promiscuous packet capture, since it's more secure compared to setuid as root. ", that is interesting, it is not working in my current scenario and we resorted to running it with the set uid bit to make it work!

I'm assuming you work on the stream application, perhaps you can get the documentation updated?

Ensure Splunk_TA_stream privileges

splunkd must be running with
root/Administrator privileges for the
streamfwd binary to capture packets on
the network interface. This applies to
both single-instance and distributed
deployments.

What you have stated does not match what the documentation says, this new mode sounds great if it would work on my particular server...I'll log a support case on that when I have some spare time, for now set uid works great with the streamfwd binary!

[vshcherbakov_sp]

it is not working in my current scenario and we resorted to running it with the set uid bit to make it work!

Would you be able to share your OS/environment config?

I'm assuming you work on the stream application, perhaps you can get the documentation updated?

Thanks for pointing this out. I created a doc ticket, we'll fix the documentation shortly.

[gjanders]

Thanks for your help.

Linux 2.6.32-431.11.2.el6.x86_64 #1 SMP Mon Mar 3 13:32:45 EST 2014 x86_64 x86_64 x86_64 GNU/Linux

Red Hat Enterprise Linux Server release 6.5 (Santiago)

[vshcherbakov_sp]

Thanks for the info, and thanks for using Stream! We'll try to reproduce the permission issue you've encounter.

[gjanders]

No problem, I've switched back to using the stuid bit again and my stream is working once again (I updated the deployment server and it removed the setuid bit and I've had it fixed again)!

[gjanders]

That is incorrect:
streamfwd: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.9, stripped
streamfwd-rhel5: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.9, stripped
streamfwd-rhel6: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, stripped

-rwxr-xr-x 1 splunk splunk 47M Nov 15 11:47 streamfwd*
-rwxr-xr-x 1 splunk splunk 47M Nov 15 11:47 streamfwd-rhel5*
-rwxr-xr-x 1 splunk splunk 48M Nov 15 11:47 streamfwd-rhel6*

However, the streamfwd did not come like that from the original file I just downloaded from splunkbase, so it was either from the previous Splunk stream version, or it was created automatically. Since the md5sum's are identical between streamfwd and streamfwd-rhel5 I'm assuming it was created automatically...

[vshcherbakov_sp]

hmm.. this doesn't look right. Here's what I see on a fresh install of Stream 7.0 on CentOS 6.5:

$ ll ~/splunk/etc/apps/Splunk_TA_stream/linux_x86_64/bin/
total 96492
lrwxrwxrwx 1 vshcherbakov wheel       15 Nov 15 17:29 streamfwd -> streamfwd-rhel5
-rwxr-xr-x 1 vshcherbakov wheel 49276856 Nov 15 17:29 streamfwd-rhel5
-rwxr-xr-x 1 vshcherbakov wheel 49528624 Nov 15 17:29 streamfwd-rhel6
[vshcherbakov@centos6 apps]$ 

[vshcherbakov_sp]

After running set_permissions.sh:

[vshcherbakov@centos6 Splunk_TA_stream]$ sudo ./set_permissions.sh 
[sudo] password for vshcherbakov: 
setting capabilities
setting setuid for streamfwd-rhel6 - linux 64 bit version
[vshcherbakov@centos6 Splunk_TA_stream]$ 
[vshcherbakov@centos6 Splunk_TA_stream]$ 
[vshcherbakov@centos6 Splunk_TA_stream]$ 
[vshcherbakov@centos6 Splunk_TA_stream]$ ll linux_x86_64/bin/
total 96492
lrwxrwxrwx 1 vshcherbakov wheel       15 Nov 15 17:29 streamfwd -> streamfwd-rhel5
-rwxr-xr-x 1 vshcherbakov wheel 49276856 Nov 15 17:29 streamfwd-rhel5
-rws--x--x 1 root         wheel 49528624 Nov 15 17:29 streamfwd-rhel6
[vshcherbakov@centos6 Splunk_TA_stream]$ 

[gjanders]

Some more checking shows this might relate to the scp to copy the files from the deployer for the search heads to the deployment server, the original server has the symlink, other servers have a binary, so perhaps the scp changed that. I'll fix it now...

Also see my other / new question now...