Archive
Highlighted

What is correct way to set-up Stream Forwarders with an Index Cluster?

Communicator

In the process of trying to get Splunk App for Stream up and running in a distributed deployment using an index cluster with 8 indexers set with repFactor = 5 and a single Stream App search-head. I have TA-stream installed on 4 forwarders. I have enabled Data Inputs > Wire Data on all 4 of these forwarders including setting the Splunk App for Stream location to the single Stream App search head (not using SSL so this is set to port 8000 using http://).

The inputs.conf file is configured on all 4 forwarders with the following settings in the [streamfwd] and [streamfwd://streamfwd] stanzas:

/opt/splunk/etc/apps/SplunkTAstream/default/inputs.conf [streamfwd]
/opt/splunk/etc/system/default/inputs.conf rcvbuf = 1572864
/opt/splunk/etc/apps/Splunk
TAstream/default/inputs.conf disabled = true
/opt/splunk/etc/system/local/inputs.conf host = ip-172-31-21-115
/opt/splunk/etc/system/default/inputs.conf index = default
/opt/splunk/etc/apps/Splunk
TAstream/default/inputs.conf source = stream
/opt/splunk/etc/apps/Splunk
TAstream/local/inputs.conf [streamfwd://streamfwd]
/opt/splunk/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunk/etc/apps/Splunk
TAstream/local/inputs.conf disabled = 0
/opt/splunk/etc/system/local/inputs.conf host = ip-172-31-21-115
/opt/splunk/etc/system/default/inputs.conf index = default
/opt/splunk/etc/apps/Splunk
TAstream/default/inputs.conf source = stream
/opt/splunk/etc/apps/Splunk
TAstream/local/inputs.conf splunkstreamapplocation = http://ip-172-31-30-208:8000/en-us/custom/splunk_app_stream/

When I try to do a search of source=stream* from the search-head I get no results. What am I missing in getting this set-up? I do see the index is pointing to default - not sure if I should be pointing to a different index. When I look at indexes on the index cluster master DMC I don't see any events in the main index.

0 Karma
Highlighted

Re: What is correct way to set-up Stream Forwarders with an Index Cluster?

Splunk Employee
Splunk Employee

What's your Splunk and App For Stream versions?

Have you verified that the forwarders are set up correctly, i.e. can you see any (non-stream) events from these forwarders in the index?

On a related note, I'd recommend enabling forwarding of the internal index from your forwarders to get diagnostic (log, stats) events from SplunkTA_Stream instances available to Splunk App for Stream (see App For Stream dashboards).

Also, have you checked splunkd.log and streamfwd.log on the forwarder machines for any errors? You may need to set up stream forwarder logging by making sure that log file location in /opt/splunk/etc/apps/SplunkTAstream/default/streamfwdlog.conf points to /opt/splunk/var/log/splunk/streamfwd.log

View solution in original post

0 Karma

Re: What is correct way to set-up Stream Forwarders with an Index Cluster?

Communicator

Splunk is 6.2.2. App for Stream is 6.2.1. Executing a search on the "dedicated" app4stream search-head (versus the search-heads in my shc) for this search:

index=internal host="ip-172-31-21-117" sourcetype="splunkapp_stream-2"

Produces this search result:

4/11/15
2:43:23.853 AM

2015-04-11 02:43:23,853 DEBUG stream:252 - DefaultDir /opt/splunk/etc/apps/framework/default/streams, LocalDir /opt/splunk/etc/apps/framework/local/streams
host = ip-172-31-21-117 source = /opt/splunk/var/log/splunk/splunkappstream.log sourcetype = splunkappstream-2

Configuration of streamfwdlog on the 4 forwarders running TA_stream:

Stream forwarder log file configuration

log4cplus.appender.streamfwdlog=log4cplus::RollingFileAppender
log4cplus.appender.streamfwdlog.layout=log4cplus::PatternLayout

The name and location for the log file

log4cplus.appender.streamfwdlog.File=/opt/splunk/var/log/splunk/streamfwd.log

Right now I have only configured TCP and UDP streams for stats only. I don't get any results returned when running the Stream Stats dashboard, e.g. Don't get any results when using any of the dashboards, for that matter.

0 Karma
Highlighted

Re: What is correct way to set-up Stream Forwarders with an Index Cluster?

Communicator

I do get the following FATAL error on the forwarders, so that can't be too good:

2015-04-10 00:55:47 INFO 139845104486208 stream.CaptureServer - Found DataDirectory: /opt/splunk/etc/apps/SplunkTAstream/data
2015-04-10 00:55:47 INFO 139845104486208 stream.CaptureServer - Found UIDirectory: /opt/splunk/etc/apps/SplunkTAstream/ui
2015-04-10 00:55:51 FATAL 139845104486208 stream.main - No found in

0 Karma
Highlighted

Re: What is correct way to set-up Stream Forwarders with an Index Cluster?

Communicator

2015-04-10 00:55:47 INFO 139845104486208 stream.CaptureServer - Found DataDirectory: /opt/splunk/etc/apps/SplunkTAstream/data
2015-04-10 00:55:47 INFO 139845104486208 stream.CaptureServer - Found UIDirectory: /opt/splunk/etc/apps/SplunkTAstream/ui
2015-04-10 00:55:51 FATAL 139845104486208 stream.main - No found in

0 Karma
Highlighted

Re: What is correct way to set-up Stream Forwarders with an Index Cluster?

Splunk Employee
Splunk Employee

You should specify the index for it to go into, as a test 'main' should be fine.

Are you running the stream capture app with root permissions, (or elevated permissions if splunk isnt running as root?) There is a script that you need to run to elevate the stream app if you are.. Check in the documentation, I dont have the link handy.

0 Karma
Highlighted

Re: What is correct way to set-up Stream Forwarders with an Index Cluster?

Communicator

I did set the streamfwd binary privileges using the setuid.sh script. IN the spirit of splunk newbieness, where do I set the configuration for forwarding to a specific index serviced in the index cluster? I am looking through the documentation for inputs.conf and outputs.conf at least up till now my lil' ol' brain has not found any documentation in the specs that call this configuration out - which config file, which stanza, which attribute.

0 Karma
Highlighted

Re: What is correct way to set-up Stream Forwarders with an Index Cluster?

Splunk Employee
Splunk Employee

Check the documentation:

http://docs.splunk.com/Documentation/StreamApp/latest/DeployStreamApp/FAQ#How_do_I_direct_traffic_fr...

Modify the file as listed, in the same manner as all inputs;

index = myindex
0 Karma
Highlighted

Re: What is correct way to set-up Stream Forwarders with an Index Cluster?

Communicator

OK. All problems now resolved with this issue. Thanks esixsplunk and vshcherbakovsplunk.

0 Karma
Highlighted

Re: What is correct way to set-up Stream Forwarders with an Index Cluster?

Communicator

Think I just found the answer to my last question but want to confirm this. Looks like the index is set in the Stream Configuration panel of the App for Stream Dashboard running on the designated search-head. Making the configuration change in this panel - I would suspect that gets pushed down to the forwarders over that "heart-beat config" channel?

Going now to confirm on this to see in terms of forwarder conf files where those settings get populated.

0 Karma