Is there a way to make forwarding/indexing decisions based on which Splunk server sent the data?

[bryanrobertson]

Is there a way to make forwarding/indexing decisions in Splunk config files based on the sending Splunk server regardless of the contents event data?

Background:

We have several Splunk servers (lets call them, A,B,C,D,E). Each server receives logs from several hosts and sources. We are need to forward data based on what Splunk server (not the host, source, or sourcetype) sent the logs.

For example:

Splunk servers A, B and C forward their logs to Splunk server D. If Splunk server D received data from Splunk server C, Splunk server D must index+forward the data to Splunk server E (regardless of the event details). Splunk server D will only index, and not forward, logs forwarded from Splunk servers A and B.

Please note: Clustering is not an option in our environment

Thank you for any feedback

[FrankVl]

I assume you are aware of the generic configuration options for routing data and such? http://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad

Basically, you will need something in the events that identifies the forwarder it came from. If you happen to have that (e.g. source value contains a file path that includes a forwarder specific piece), you could use that for the filtering, but I would expect you probably have some data sources where you do not have such identifying info in the event?

One solution you could look at is configure multiple splunktcp inputs (separate ports) on forwarder D, each with its own related forwarding and indexing config and then send A and B to one input and C to the other. But while that is a nice idea with a handful of forwarders like your example, this could become a bit complicated if the real situation is much more complex.

Alternatively you may just have to rethink your architecture and not send data from C to D in the first place, but get it to E directly or through another intermediate forwarder F.

[bryanrobertson]

Thank you. These are all good ideas. I have read over the “Route and filter data” and I use props.conf and transforms.conf in other places for routing purposes. The event data is so different in some cases that there are no easily repeatable patterns to make routing decisions. In the long term I am going to make the data easier to identify (separate indexes or create custom fields at index time), but all of that would require a lot of work for our setup.
I am going to try the multiple splunktcp inputs next week as that sounds like the easiest for our environment.

[FrankVl]

Alright, good luck! Looking forward to hear if you succeeded 🙂

[David_Naylor]

Hi Bryanrobertson,

Yes you can specify _INDEX_AND_FORWARD_ROUTING or _TCP_ROUTING under the following stanza

See the inputs.conf doc for more details on that stanza

Hope this helps, goodluck! - David

[FrankVl]

How exactly will that help his need to make routing decisions based on which upstream forwarder was sending?

[David_Naylor]

My theory was, that on the intermediate forwarder's input.conf you can specify the following stanza

• Forwarder-specific settings for splunktcp.

• Receivers use this input stanza.

• This is the same as the [tcp://] stanza, except the remote server is assumed
  to be a Splunk instance, most likely a forwarder.

Under that stanza they should be able to specify forwarder specific settings for _INDEX_AND_FORWARD_ROUTING or _TCP_ROUTING parameters to route the events on a per forwarder basis.

[bryanrobertson]

Thank you. After reading over the inputs.conf specs file more, I think grouping the forwarders under the splunktcp stanza and using TCP_ROUTING will work for our case. I will give it a go next week

[FrankVl]

Unless you suggest to create separate splunktcp inputs for each (group of) forwarder(s), my question is: how to "specify forwarder specific settings for _INDEX_AND_FORWARD_ROUTING or _TCP_ROUTING parameters"?