Why is Splunk App for Stream only starting with 1 of 3 universal forwarders when all have the same configurations deployed?

[f_luciani]

Splunk Enterprise/Universal Forwarder 6.1.2 App for Stream 6.0.2
I have an indexer/search head and 3 forwarders. All 3 forwarders have App for Stream deployed, but only one has it started. I have been comparing the configurations of all 3 but can't seem to find any discrepancies. Nevertheless, only in 1 of the 3, App for Stream starts with the forwarder, the other 2 only have the forwarder started, the App for Stream won't start when I start the forwarder. The app was deployed the same way in all 3 forwarders; I compared the configuration files and, apart from the bits that are host-specific, all configs look the same. What am I missing here? Where should I start looking for clues?

[mdickey_splunk · Accepted solution]

What operating systems are the forwarders running on (all the same)? Do the network devices all have the same name? Is the "streamfwd" process running on the "bad" forwarders, and are you able to access the web UI on port 8889 (http://<server>:8889)?

Please note that "App for Stream" includes both an app -- used for centralized configuration and reporting and which would normally be deployed on your search head -- and a technology add-on or TA. The app is located in $SPLUNK_HOME/etc/apps/splunk_app_stream and the TA is in $SPLUNK_HOME/etc/apps/Splunk_TA_stream. The TA is the one you should distribute to your universal forwarders, and on all forwarders the "splunk_stream_app_location" configuration parameter in your inputs.conf file should point to the same location, which is where the app is installed and running.

[f_luciani]

One more thing: I've just ran a tail -f on splunkd.log and got this after starting Splunk on both faulty forwarders:

10-07-2014 15:57:29.511 -0300 ERROR ModularInputs - Introspecting scheme=streamfwd: script running failed (killed by signal 8: Floating point exception).
10-07-2014 15:57:29.511 -0300 ERROR ModularInputs - Unable to initialize modular input "streamfwd"  defined inside the app "Splunk_TA_stream": Introspecting scheme=streamfwd: script running failed (killed by signal 8: Floating point exception).

[mdickey_splunk]

SUSE is not one of our supported operating systems, and the ERROR messages indicate that our binary will not run on those platforms.

[f_luciani]

Hi, mdickey_splunk,

Thanks for your answer. The faulty forwarders are heavily customized versions of Debian and SuSE Enterprise:

• SUSE Linux Enterprise Server 10 - 2.6.16.21-0.8-bigsmp (geeko@buildhost) (gcc version 4.1.0 (SUSE Linux)) #1 SMP Mon Jul 3 18:25:39 UTC 2006
• Debian 5.3.1 - 3.0.13-0.27-default #1 SMP Wed Feb 15 13:33:49 UTC 2012 (d73692b) x86_64 GNU/Linux

The network devices are the same for all servers - eth0.

Can't access the admin interface on port 8889 due to client's firewall restrictions, they are adamant about this.

The search head is fine, both app and add-on are there, the fully working forwarder reports there (it collects logs and wire data, plus it has the same issue the search head has, meaning, the app in both will not stop when I stop splunk, I have to kill the process manually, see question 154783 - link not working - for more details). They can be found in the default folders inside the search head:

• /opt/splunk/etc/apps/Splunk_TA_stream
• /opt/splunk/etc/apps/splunk_app_stream

The add-on has been configured via Splunk web; after that I installed the app, it has been working since then. I've used Deployment Monitor to deploy App for Stream in all forwarders.

The inputs.conf for the app in all forwarders has the same content. The faulty forwarders are working, they send log data, but App for Stream never worked for them.

Any ideas?