What is correct way to set-up Stream Forwarders with an Index Cluster?

[transtrophe]

In the process of trying to get Splunk App for Stream up and running in a distributed deployment using an index cluster with 8 indexers set with repFactor = 5 and a single Stream App search-head. I have TA-stream installed on 4 forwarders. I have enabled Data Inputs > Wire Data on all 4 of these forwarders including setting the Splunk App for Stream location to the single Stream App search head (not using SSL so this is set to port 8000 using http://).

The inputs.conf file is configured on all 4 forwarders with the following settings in the [streamfwd] and [streamfwd://streamfwd] stanzas:

/opt/splunk/etc/apps/Splunk_TA_stream/default/inputs.conf [streamfwd]
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunk/etc/apps/Splunk_TA_stream/default/inputs.conf disabled = true
/opt/splunk/etc/system/local/inputs.conf host = ip-172-31-21-115
/opt/splunk/etc/system/default/inputs.conf index = default
/opt/splunk/etc/apps/Splunk_TA_stream/default/inputs.conf source = stream
/opt/splunk/etc/apps/Splunk_TA_stream/local/inputs.conf [streamfwd://streamfwd]
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunk/etc/apps/Splunk_TA_stream/local/inputs.conf disabled = 0
/opt/splunk/etc/system/local/inputs.conf host = ip-172-31-21-115
/opt/splunk/etc/system/default/inputs.conf index = default
/opt/splunk/etc/apps/Splunk_TA_stream/default/inputs.conf source = stream
/opt/splunk/etc/apps/Splunk_TA_stream/local/inputs.conf splunk_stream_app_location = http://ip-172-31-30-208:8000/en-us/custom/splunk_app_stream/

When I try to do a search of source=stream* from the search-head I get no results. What am I missing in getting this set-up? I do see the index is pointing to default - not sure if I should be pointing to a different index. When I look at indexes on the index cluster master DMC I don't see any events in the main index.

[nickstone]

did this ever get solved?

I have the same error message running Stream 7.1 on a Centos7 (rhel) box.

The truncation of the logs (forum formatting) should look something like this:

2018-05-13 08:28:38 FATAL [140300864640896] (main.cpp:1150) stream.main - Failed to start streamfwd, the process will be terminated: No <stanza> found in <configuration>

[vshcherbakov_sp · Accepted solution]

What's your Splunk and App For Stream versions?

Have you verified that the forwarders are set up correctly, i.e. can you see any (non-stream) events from these forwarders in the index?

On a related note, I'd recommend enabling forwarding of the _internal index from your forwarders to get diagnostic (log, stats) events from Splunk_TA_Stream instances available to Splunk App for Stream (see App For Stream dashboards).

Also, have you checked splunkd.log and streamfwd.log on the forwarder machines for any errors? You may need to set up stream forwarder logging by making sure that log file location in /opt/splunk/etc/apps/Splunk_TA_stream/default/streamfwdlog.conf points to /opt/splunk/var/log/splunk/streamfwd.log

[transtrophe]

Splunk is 6.2.2. App for Stream is 6.2.1. Executing a search on the "dedicated" app4stream search-head (versus the search-heads in my shc) for this search:

index=_internal host="ip-172-31-21-117" sourcetype="splunk_app_stream-2"

Produces this search result:

4/11/15
2:43:23.853 AM

2015-04-11 02:43:23,853 DEBUG stream:252 - DefaultDir /opt/splunk/etc/apps/framework/default/streams, LocalDir /opt/splunk/etc/apps/framework/local/streams
host = ip-172-31-21-117 source = /opt/splunk/var/log/splunk/splunk_app_stream.log sourcetype = splunk_app_stream-2

Configuration of streamfwdlog on the 4 forwarders running TA_stream:

Stream forwarder log file configuration

log4cplus.appender.streamfwdlog=log4cplus::RollingFileAppender
log4cplus.appender.streamfwdlog.layout=log4cplus::PatternLayout

The name and location for the log file

log4cplus.appender.streamfwdlog.File=/opt/splunk/var/log/splunk/streamfwd.log

Right now I have only configured TCP and UDP streams for stats only. I don't get any results returned when running the Stream Stats dashboard, e.g. Don't get any results when using any of the dashboards, for that matter.

[transtrophe]

I do get the following FATAL error on the forwarders, so that can't be too good:

2015-04-10 00:55:47 INFO 139845104486208 stream.CaptureServer - Found DataDirectory: /opt/splunk/etc/apps/Splunk_TA_stream/data
2015-04-10 00:55:47 INFO 139845104486208 stream.CaptureServer - Found UIDirectory: /opt/splunk/etc/apps/Splunk_TA_stream/ui
2015-04-10 00:55:51 FATAL 139845104486208 stream.main - No found in

[transtrophe]

2015-04-10 00:55:47 INFO 139845104486208 stream.CaptureServer - Found DataDirectory: /opt/splunk/etc/apps/Splunk_TA_stream/data
2015-04-10 00:55:47 INFO 139845104486208 stream.CaptureServer - Found UIDirectory: /opt/splunk/etc/apps/Splunk_TA_stream/ui
2015-04-10 00:55:51 FATAL 139845104486208 stream.main - No found in

[esix_splunk]

You should specify the index for it to go into, as a test 'main' should be fine.

Are you running the stream capture app with root permissions, (or elevated permissions if splunk isnt running as root?) There is a script that you need to run to elevate the stream app if you are.. Check in the documentation, I dont have the link handy.

[transtrophe]

I did set the streamfwd binary privileges using the setuid.sh script. IN the spirit of splunk newbieness, where do I set the configuration for forwarding to a specific index serviced in the index cluster? I am looking through the documentation for inputs.conf and outputs.conf at least up till now my lil' ol' brain has not found any documentation in the specs that call this configuration out - which config file, which stanza, which attribute.

[transtrophe]

Think I just found the answer to my last question but want to confirm this. Looks like the index is set in the Stream Configuration panel of the App for Stream Dashboard running on the designated search-head. Making the configuration change in this panel - I would suspect that gets pushed down to the forwarders over that "heart-beat config" channel?

Going now to confirm on this to see in terms of forwarder conf files where those settings get populated.

[vshcherbakov_sp]

You can add index = your_index to the inputs.conf stanza to overwrite the default index for Stream Forwarder events. The index field in the Stream configuration UI (on the search head) overrides the default or inputs.conf value on a per-stream base.

[vshcherbakov_sp]

BTW, were you able to resolve the FATAL error problem in streamfwd.log? Not sure what the actual issue is since the error message you posted seems to be truncated.. If it's still an issue, please post the full error message to give us a better idea of what's going on.

[esix_splunk]

Check the documentation:

http://docs.splunk.com/Documentation/StreamApp/latest/DeployStreamApp/FAQ#How_do_I_direct_traffic_fr...

Modify the file as listed, in the same manner as all inputs;

index = myindex

[transtrophe]

OK. All problems now resolved with this issue. Thanks esix_splunk and vshcherbakov_splunk.