Knowledge Management

What is correct way to set-up Stream Forwarders with an Index Cluster?

transtrophe
Communicator

In the process of trying to get Splunk App for Stream up and running in a distributed deployment using an index cluster with 8 indexers set with repFactor = 5 and a single Stream App search-head. I have TA-stream installed on 4 forwarders. I have enabled Data Inputs > Wire Data on all 4 of these forwarders including setting the Splunk App for Stream location to the single Stream App search head (not using SSL so this is set to port 8000 using http://).

The inputs.conf file is configured on all 4 forwarders with the following settings in the [streamfwd] and [streamfwd://streamfwd] stanzas:

/opt/splunk/etc/apps/Splunk_TA_stream/default/inputs.conf [streamfwd]
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunk/etc/apps/Splunk_TA_stream/default/inputs.conf disabled = true
/opt/splunk/etc/system/local/inputs.conf host = ip-172-31-21-115
/opt/splunk/etc/system/default/inputs.conf index = default
/opt/splunk/etc/apps/Splunk_TA_stream/default/inputs.conf source = stream
/opt/splunk/etc/apps/Splunk_TA_stream/local/inputs.conf [streamfwd://streamfwd]
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunk/etc/apps/Splunk_TA_stream/local/inputs.conf disabled = 0
/opt/splunk/etc/system/local/inputs.conf host = ip-172-31-21-115
/opt/splunk/etc/system/default/inputs.conf index = default
/opt/splunk/etc/apps/Splunk_TA_stream/default/inputs.conf source = stream
/opt/splunk/etc/apps/Splunk_TA_stream/local/inputs.conf splunk_stream_app_location = http://ip-172-31-30-208:8000/en-us/custom/splunk_app_stream/

When I try to do a search of source=stream* from the search-head I get no results. What am I missing in getting this set-up? I do see the index is pointing to default - not sure if I should be pointing to a different index. When I look at indexes on the index cluster master DMC I don't see any events in the main index.

0 Karma
1 Solution

vshcherbakov_sp
Splunk Employee
Splunk Employee

What's your Splunk and App For Stream versions?

Have you verified that the forwarders are set up correctly, i.e. can you see any (non-stream) events from these forwarders in the index?

On a related note, I'd recommend enabling forwarding of the _internal index from your forwarders to get diagnostic (log, stats) events from Splunk_TA_Stream instances available to Splunk App for Stream (see App For Stream dashboards).

Also, have you checked splunkd.log and streamfwd.log on the forwarder machines for any errors? You may need to set up stream forwarder logging by making sure that log file location in /opt/splunk/etc/apps/Splunk_TA_stream/default/streamfwdlog.conf points to /opt/splunk/var/log/splunk/streamfwd.log

View solution in original post

0 Karma
Get Updates on the Splunk Community!

Join Us for Splunk University and Get Your Bootcamp Game On!

If you know, you know! Splunk University is the vibe this summer so register today for bootcamps galore ...

.conf24 | Learning Tracks for Security, Observability, Platform, and Developers!

.conf24 is taking place at The Venetian in Las Vegas from June 11 - 14. Continue reading to learn about the ...

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...