Splunk Stream Distributed Indexing

[jwelters]

We are looking at deploying splunk stream to replace some current monitoring solutions.

One question I ran into was:
How do you configure splunk stream on a heavy forwarder to forward in a distributed manner to multiple indexers ?

We are running the most recent version of splunk and intend to use distributed indexing to 5 indexers.

The docs list it as a deployment option however, there isn't any clear documentation I could find concernnig how to do this.

[mdickey_splunk]

The streamfwd binary included in the App for Stream TA (Splunk_TA_stream) is a modular input. Like any other modular input, it runs as a subprocess of splunkd and writes all events to stdout. It is completely up to splunkd (your heavy or universal forwarder) where and how the data is sent to indexers. Stream plays no part in it.

[jwelters]

Perhaps I've asked the question the wrong way.

We know how to send data to the distributed indexing environment by placing the appropriate stanza in outputs.conf. The splunk stream configuration does not adhere to this configuration setting. Your'e right in this case heavy / universal probably doesn't matter in our case we are using the heavy forwarders for other reasons on these particular hosts.

[csharp_splunk]

Why do you say that? Data should go where your outputs.conf says to go.

[jwelters]

Only because testing has shown that while all of the other data flows properly, splunk stream data doesn't seem to distribute.

When you configure the input from wire data you list a destination, we currently have that set to the same heavy forwarder in hopes that would result in the forwarder sending the information through the normal routing in outputs.conf. That doesn't work.

I can also point the splunk app for stream location to be one of the indexers and that also works fine.

The goal would be to configure it in the way that both you and I expect it to. So the outputs.conf drives where the data goes, and the stream forwarder just captures it to route.

These are all systems that have well functioning configurations in place other then the stream functionality. That is it gathers a whole boat load of logs and sends to distributed indexers and everything has worked amazingly well. Add stream app, enable TCP/UDP, and add in the wire data inputs. The wire data input is currently set the default which I would expect to work, then just use the underlying outputs.conf to send the data to the distributed indexer environment.

[gkanapathy]

What do you mean by "When you configure the input from wire data you list a destination"? Do you mean you created a new wire data instance, and pointed the Splunk App for Stream Location at your heavy forwarder? Or do you mean you modified the default mod input instance and added a _TCP_ROUTING entry that pointed at the heavy forwarder?

[gkanapathy]

A Universal or Light Forwarder can also forward to multiple indexers, so if that is the only thing you are considering, there is no point in using a Heavy Forwarder. Nevertheless, there is no difference in deploying Stream on Heavy Forwarder vs a Universal Forwarder.