Why am I not getting data from the Splunk App for Stream using a universal forwarder with my current configuration?

[ilennynzx]

I'm having problems with getting Splunk_TA_stream to push data from a universal forwarder to an indexer. In short I've copied the Splunk_TA_stream folder into the apps folder on a universal forwarder and configured it as best I can tell, to send data to an indexer. However none of the data from the forwarder is showing up in the indexer. I need clear information on how to setup Splunk_TA_stream on a universal forwarder and how to ensure the Indexer receives that data.

The current setup has a full splunk install which is running as an indexer and search head, the install is version 6.2.0 with app for stream 6.1.0.
There are several servers with the splunk unversal forwarder setup to feed data to the indexer. The forwarder I'm testing with is version 6.1.3 with the Splunk_TA_stream folder from app for stream 6.1.0.

On the indexer Wire Data has been enabled with name streamfwd and source "http://localhost:8000/en-us/custom/splunk_app_stream/". No other configuration has been done on the search head; it is successfully capturing wire data locally.
The universal forwarder has been configured to forward to the indexer and has successfully been tested by following several application logs and putting them into a test index. Splunk_TA_stream inputs.conf on the forwarder has been configured as follows:

[streamfwd://streamfwd]
splunk_stream_app_location = http://INDEXER_FQDN:8000/en-us/custom/splunk_app_stream/
disabled = 0

Where INDEXER_FQDN is the full domain name of the splunk indexer. There are no errors in the streamfwd.log for that configuration, however if I change the path for app_location in the config (say replace /custom/ with /customBOB/) or if I stop the splunk indexer I get "stream.CaptureServer - Unable to ping server".
Based on that it appears that streamfwd on the universal forwarder is communicating with the splunk indexer at some level.

There have been issues with firewalls and routing between the two servers, at this time the known splunk ports are allowed between the two servers and data indexes correctly when configured in inputs.conf. I have to assume that I've messed up or missed a configuration setting somewhere.
Can anyone confirm exactly how the stream config is supposed to be setup on a universal forwarder and how the indexer is configured for each streamfwd source?

[dailv1808]

I have save problem, so how to fix this?

[ilennynzx · Accepted solution]

I think I've figured it out. It appears the Splunk_TA_Stream app inherited the default index from another app installed on the forwarder. While checking which hosts were feeding into the custom app I was working on using metadata hosts, I got an undefined host ($decidehostonstartup) which contained stream data.
I'd mistakenly deleted the local inputs.conf which had the host defined in it on one of the forwarders I was working on. That caused the erroneous host to show up for the stream data app. Begs the question why would the Splunk_TA_Stream app take the index defined in another app instead of the default/inputs.conf but at least I'm getting data now.
Have re-enabled stream on the server I was originally testing with and got the same result, if undefined it uses the index from my custom app.

Docs for Stream suggest source=stream* as a search for data, try adding index=* to the start of the search if you get no results.

Now to figure out what the hell with the indexes.

[mdickey_splunk]

It's unnecessary to run streamfwd (or the "Wire Data" Data Input) on your indexer. This will only capture events that the indexer sees. I'm stumped as it sounds like you have everything setup and working OK. What is the search you are using to find stream events? "source=stream*" should work.

[ilennynzx]

Yeah, I'd turned it on to make sure app was capturing data and that I my search was returning results. It confirmed that streamfwd was working locally.
I am pretty certain that the forwarder is capturing data from the information on the stream management page on the forwarder. At this stage I'm going to look at network changes between the indexer and the forwarder to completely rule out the firewall.

[ilennynzx]

Further tests: I disabled the localhost streamfwd on the indexing server so it was no longer receiving local stream data. The management page on the forwarder still showed the same results, events being captured and the indexer as the app location.
So it I guess it's capturing data locally and attempting to send to the indexer.

[ilennynzx]

I've since removed Splunk_TA_stream from the forwarder and configured it to use the indexer as a deployment server. I then got the indexer to deploy the stream TA to the forwarder, it's resulted in the same configuration as my manual setup and same results so far.

[mdickey_splunk]

I believe your splunk_stream_app_location is configured correctly, otherwise you would get those "Unable to ping server" errors. Are you running splunkd as root, or have you run the setuid.sh script to ensure that it will run streamfwd as root? The streamfwd process must be running as root; otherwise you will not be able to see any traffic. Also you should be able to verify it's capturing data using the web interface at http://FORWARDER_FQDN:8889

[ilennynzx]

Forwarder is running as user splunk, setuid.sh has been run and the streamfwd binary is running as root.
The web interface appears to be showing it capturing data and it has the indexer host name in it, actually it looks like it is showing the data capture events for the indexer. I get the feeling I've got the configuration backwards.

*clarification: The forwarder is showing a number for data capture events, then below that it has a separate box containing the URL for the indexer and port 8000 plus the same number for data capture as above. It reads as though the forwarder is receiving the data capture events from the indexer.

[okrabbe_splunk]

Did you get the stream app location from the example in the indexer? I think it is wrong and a bad example.

Try something like this instead

[streamfwd://streamfwd]
splunk_stream_app_location = http://INDEXER_FQDN:8000/
disabled = 0

[ilennynzx]

Changed app_location as suggested and it made no difference at all. No ping errors in the streamfwd logs but no data in the indexer either. I've reverted it to the full path for the time being, but can keep that change in mind as I try other things with the configuration.