Splunk Stream: Why is there inconsistent data produced between the deployment server and heavy forwarder when running streamfwd?

[mathiask]

I am getting inconsistent issues when running the streamfwd on CentOS 7.x
On the Deployment server some data is captured, i.e. Stream Estimate shows statistics
The heavy forwarders, which are generally setup the same way, do not produce any data

Setup:

• CentOS 7.1 Systems cat /etc/redhat-release CentOS Linux release 7.2.1511 (Core)
• Splunk Enterprise 6.5 on Deployment Server and 2 Heavy forwarders
• Splunk is running with the user splunk:splunk, not root

Step 1: Installing Splunk Stream on the Deploymentserver, go to app directory ./set_permissions
Step 2: Deploy App, go to forwarders, ./set_permission

Now the deployment server and forwarders should set up the same way.
But on the forwarder I get the following message
SnifferReactor failed to open pcap adapter for device <eth0>. Error message:

When the forwarder is run as root, which is not an option long term, then it works the same

I first thought the permissions might be not set correctly as
splunk 4212 0.5 1.7 631520 68836 ? Ssl 17:42 0:00 /opt/splunk/etc/apps/Splunk_TA_stream/linux_x86_64/bin/streamfwd
actually calls a reference of the rhel5 version on the Deployment server
lrwxrwxrwx. 1 splunk splunk 15 Nov 25 17:29 streamfwd -> streamfwd-rhel5
-rwxr-xr-x. 1 splunk splunk 47M Nov 5 07:28 streamfwd-rhel5
-rws--x--x. 1 root splunk 48M Nov 5 07:28 streamfwd-rhel6
On the forwarder it actually calls a binary instead, which is identical to rhel5
-rwxr-xr-x. 1 splunk splunk 47M Nov 25 19:00 streamfwd
-rwxr-xr-x. 1 splunk splunk 47M Nov 25 19:00 streamfwd-rhel5
-rws--x--x. 1 root splunk 48M Nov 25 19:00 streamfwd-rhel6
This might be because the deployment app is set up like this and it deploys the referenced binary instead of the link
lrwxrwxrwx. 1 splunk splunk 15 Nov 25 17:29 streamfwd -> streamfwd-rhel5

But this does neither to explain

• why are the permissions "fixed" for rhel6 when rhel5 is actually called?
• why does it work on the deployment server but not on the heavy forwarder?

[mathiask · Accepted solution]

Thanks

So in summary

• The main issue is caused because on the deployment server the streamfwd binary is referenced but during the deployment process this will cause the actual binary to be copied instead of the reference.
• A fix is in the works.
• When the binary is replaced by the reference it works. Until then it is probably best to deploy the App via some other means, otherwise Splunk deployment will overwrite the reference again.
• The linux streamfwd binary has the appendix "rhel5" because RHEL 5 is the lowest supported version for stream
• The RHEL 6 version is for 10 Gbps because of the used Intel DPDK framework kernel requirement

Does this mean that currently "only" 10 Gbps is supported?
I just ask because of curiosity, not because I seriously intend to hook up our 100 Gbps links to it 🙂

[vshcherbakov_sp]

Does this mean that currently "only" 10 Gbps is supported?

DPDK technically supports 40Gbps cards, but we haven't yet tested it..

[mathiask]

Update

I renamed the binary on the forwarder and then created a link like on the deployment server
It now works in the same way

Conclusion
The deployment server screws it up when the Splunk_TA_stream is deployed
How can this be fixed, because with the next deployment cycle the local configuration will be wiped out, again.
I rather prefer to use the on board tools rather than for some and for some others I will use Ansible et. al.

Still, why is the rhel5 version referenced on an "RHEL 7" system ?

[mathiask]

I actually find it a bit confusing that it works after I replace the binary with the reference
Because rhel6 seems to have the "right" permissions and not rhel5
-rws--x--x. 1 root splunk 48M Nov 25 19:00 streamfwd-rhel6
And if I check the set_permissions.sh this is what it actually does, or am I wrong?

[vshcherbakov_sp]

set_permissions.sh scripts gives different permissions to rhel5 and rhel6 binaries: it sets linux capabilities to rhel5 and root permissions (setuid) to rhel6. This is because the "regular" libpcap-based capture (rhel5) doesn't need full root permissions to capture network packets on modern linux, while DPDK still requires root (or we haven't figured out how to make it run in non-root mode)

[vshcherbakov_sp]

@mathiask,

Sorry about the troubles.. We're working on a fix for this issue.

Still, why is the rhel5 version referenced on an "RHEL 7" system ?

Yeah, that's a bit confusing: rhel5 and rhel6 suffixes denote the lowest supported release versions for each binary. Stream generally supports RHEL5 (and its contemporaries from other distros) and up; the reason why rhel6 binary exists is because Stream also needs to support scalable data capture on a 10Gbps SPAN/TAP port, and Intel DPDK framework we use for that task requires newer kernel version, etc.