All Apps and Add-ons

How to setup Splunk App for Stream in a Distributed environment?

Contributor

We have a setup with multiple Search Heads, Indexers, Universal Forwarders, Heavy Forwarders.
we are trying to setup Splunk-app-for-Stream to collect stream data from Universal Forwarders into Search Head.
we have gone through tons of documentation, no use. There is no proper documentation for this scenario.
we understand that we needed to install Splunk app for stream on Search Head and Splunk_TA_Stream on the Forwarder and indexer as well.
we followed the following steps :
1. Download splunk-app-for-stream_642 from SplunkBase and install on SH
2. Download Splunk_TA_Stream on Univforwarder and enable wiredata input
3. Configure Splunk_TA_stream/local/inputs.conf
• Changes done in forwarder.
-bash-4.1$ cat /opt/splunkforwarder/etc/system/local/outputs.conf
[tcpout]
forwardedindex.2.whitelist = (_audit|_introspection|_internal)
. Input.conf
[streamfwd://streamfwd]
splunk_stream_app_location = https://FQDN-of-SearchHead:8000/splunk/en-US/app/splunk_app_stream/
disabled = 0

Now I don't have any idea on what needs to be done on Indexer?
Anyone who has a clear steps on this type of configuration please help !!!
currently I don't see any data with source=stream* coming into SH!!!

1 Solution

Splunk Employee
Splunk Employee

I believe you have an invalid splunk_app_stream_location. It should look like this:
splunk_stream_app_location = http://FQDN-of-SearchHead:8000/en-us/custom/splunk_app_stream/

You can try running this from one of your universal forwarders to verify that it's correct and able to communicate with the app.
curl http://FQDN-of-SearchHead:8000/en-us/custom/splunk_app_stream/ping/

You should get a JSON response like this:
{"dateLastUpdated": 1458251157984, "version": "6.4.2"}

It's also not clear that your universal forwarders are configured to send events to your indexers via outputs.conf. You can find documentation on this at http://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Configureforwarderswithoutputs.confd.

View solution in original post

Contributor

Finally we resolved the issue when we corrected the URI as suggested.
My advice is to check the Wbaccess.log to know check if the SH is responding with proper reply to UF.
Actually it was a simple configuration steps, we spent almost a week to figure out.

I would place a request for updating the Splunk-app-for-stream documentation to be more clear on distributed setup.

0 Karma

Splunk Employee
Splunk Employee

I believe you have an invalid splunk_app_stream_location. It should look like this:
splunk_stream_app_location = http://FQDN-of-SearchHead:8000/en-us/custom/splunk_app_stream/

You can try running this from one of your universal forwarders to verify that it's correct and able to communicate with the app.
curl http://FQDN-of-SearchHead:8000/en-us/custom/splunk_app_stream/ping/

You should get a JSON response like this:
{"dateLastUpdated": 1458251157984, "version": "6.4.2"}

It's also not clear that your universal forwarders are configured to send events to your indexers via outputs.conf. You can find documentation on this at http://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Configureforwarderswithoutputs.confd.

View solution in original post

Splunk Employee
Splunk Employee

Hi bkumarm,

As of Stream 6.4.2, you'll need the following layout...

  • splunk_app_stream : deployed to a single search head, this is your primary configuration point for Stream, as well as viewing the dashboards.
  • Splunk_TA_stream: needs to be deployed to any splunk instance that will perform searching, parsing, or local data capturing. In your scenario, this means the search head, forwarders and indexers. On the search head and indexers, if you do NOT want to have local capture, then make sure the copy they get have disabled=1 in their inputs.conf.

On the systems actually performing data capture (the forwarders), if they are Linux, you'll have one additional step, which is to allow the streamfwd process the ability to perform network capture. The docs list that out here: http://docs.splunk.com/Documentation/StreamApp/6.4.2/DeployStreamApp/InstallSplunkAppforStream#Step_...

One last thing, out of the box, the TA understands certain network interface naming conventions. If you're on Linux, that defaults to eth# and en#. If you happen to use a different naming convention, you'll need to make a copy of Splunk_TA_stream/default/streamfwd.xml into Splunk_TA_stream/local/streamfwd.xml and make it look similar to this (assuming your interface is of the form enp#s#):

<CmConfig xmlns="http://purl.org/cloudmeter/config" version="6.4.2">
  <Port>8889</Port>
  <UIDirectory>../ui</UIDirectory>
  <DataDirectory>../data</DataDirectory>
  <LogConfig>streamfwdlog.conf</LogConfig>
  <Capture>
    <InterfaceRegex>enp[0-9]s[0-9]</InterfaceRegex>
  </Capture>
</CmConfig>

For Windows, you'll want to specify the network alias, such as "Local Area Network", or whatever your naming convention happens to be.

You'll need to restart the splunk instance after making any of these changes.

Hope that helps,
-js