Splunk Stream: Why does my heavy forwarder not show up in the Distributed Forwarder Management?

[wweiland]

Hi,

I'm about to pull what little hair I have left out. I have a SH and Indexer Cluster running 6.5.1. My cluster uses our own SSL certs for server.conf, web.conf, and inputs.conf, which appear to be working fine. I've installed Splunk Steam (splunk_app_stream and Splunk_TA_stream) on my deployment/admin server. I've installed Splunk_TA_stream on my indexers and a heavy forwarder. I set the location of my server running the splunk_app_stream in the inputs.conf and the Splunk_TA_stream on the heavy forwarder. My problem is that the heavy forwarder still does not show up in the Distributed Forwarder Manager even though I see 2 way traffic via tcpdump. Can anyone help me who has set this up before? What information do you need?

Thank you so much in advance,
Todd

[vshcherbakov_sp · Accepted solution]

Hi @wweiland,

Sorry to hear about your troubles with Stream..

What OS is your heavy forwarder running on? What's the Stream forwarder config there? Have you run ./set_permissions.sh script (assuming it's *nix)?

Do you have anything suspicious in $SPLUNK_HOME/var/log/splunk/streamfwd.log file on the heavy forwarder? Do you have _internal index on heavy being forwarded from HFW to IDX?

[wweiland]

It appears that there is stuff in _internal that is absolutely necessary for this app to work properly. I had not set it to forward to the indexers yet.

Thanks

[gawilliams]

Wow thanks for this input! We had to use our Heavy Forwarder to manage stream configurations because we have a Search Head cluster which doesn't support global tokens (as far as we can tell). Once we turned on and configured distributed search on our Heavy Forwarder to the indexer cluster, the Stream Forwarder Management started working! Wish they would document this in the Splunk docs (as far as I can tell it's not called out).

[gjanders]

gawilliams on any documentation page you can hit the submit feedback button and they will usually update the documentation !

[coltwanger]

What was in _internal that was necessary? I am forwarding _internal from all of my hosts, but I am experiencing the same issues where my forwarders do not show up under Distributed Forwarder Management. 6.5.1 environment also.

[vshcherbakov_sp]

check if you're getting sourcetype="stream:stats" events in the _internal index on the SH - this is what Stream UI requires

[coltwanger]

Thanks for the response! I actually found out I had an issue in some CONF that was preventing me from accessing the endpoint, but all is well now 🙂