Why is Splunk App for Stream not restarting with Splunk?

[f_luciani]

Hi,

Not sure if it is really an issue or if I just skipped some important bit of documentation. Thing is, every time I issue a stop to Splunk, if I check with "ps -ef | grep sp" I notice that the following process fails to stop and keeps running:

root     15080     1  2 09:48 ?        00:02:28 /opt/splunk/etc/apps/Splunk_TA_stream/linux_x86_64/bin/streamfwd

... which forces me to kill the App for Stream process manually and then start Splunk. The app then starts normally but if I stop Splunk again, the same happens again and I have to repeat the procedure (manually stop the app so it can be started when Splunk starts). If I do not kill the process, App for Stream kinda loses its connection to Splunk and I can't see data streaming in until I kill the process and stop/start or simply restart Splunk.

Is this an issue or should I consider this the normal app behaviour?

[csharp_splunk]

Can you give us details on your OS/distribution please?

[f_luciani]

@csharp_splunk:

I am not sure if the info below helps (extracted from /proc/version):

• Linux version 2.6.26-2-xen-amd64 (Debian 2.6.26-26lenny3) (dannf@debian.org) (gcc version 4.1.3 20080704 (prerelease) (Debian 4.1.2-25)) #1 SMP Mon Jun 13 18:44:16 UTC 2011

• Linux version 3.2.0-58-generic (buildd@allspice) (gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #88-Ubuntu SMP Tue Dec 3 17:37:58 UTC 2013

...since I am running both Enterprise and UniversalForwarder inside proprietary appliances built around popular distros (Ubuntu and Debian). In theory, they should behave more or less like any other vanilla install of Debian or Ubuntu, but I cannot say for sure since I wasn't part of the team responsible for this customization, thus cannot guarantee. All I can say for sure is that, upon login, I can see the Ubuntu machine's greetings, which include version info (12.04 LTS - it is already offering a distro-upgrade to 14-04 LTS).

[mdickey_splunk]

What version of Splunk and App for Stream are you using? Is there anything in your splunkd.log or streamfwd.log files (both should be located in $SPLUNK_HOME/var/log/splunk) that indicates why it may not be shutting down?

[f_luciani]

Splunk 6.1.2 and app 6.0.1, logs don't show anything conclusive. There are 2 lines in streamfwd.log that appear after Splunk is restarted:

ERROR 140271892133696 pion.http.server - Unable to bind to port 8889: bind: Address already in use
FATAL 140271892133696 stream.main - bind: Address already in use

The app keeps running after Splunk is stopped:

ERROR 139920218298112 stream.CaptureServer - Unable to ping server (Unable to establish connection to localhost: Connection refused): 9ed0e0cb-0cd2-465e-8fb0-fd7771c98fa7

[mathiask]

Hi
We observed the same behaviour. We either had to restart the system or restart the process separately.
I think this normal in the current version, but expect that in a future version the streamfwd process can be controlled through Splunk directly.
I also didn't manage to restart the process though the streamfwd UI on port 8889, I got some kind of PUT permission error.
I suspect that longterm all this will be integrated directly to splunk/splunk UI.