Getting Data In

Is there a way to make forwarding/indexing decisions based on which Splunk server sent the data?

bryanrobertson
New Member

Is there a way to make forwarding/indexing decisions in Splunk config files based on the sending Splunk server regardless of the contents event data?

Background:

We have several Splunk servers (lets call them, A,B,C,D,E). Each server receives logs from several hosts and sources. We are need to forward data based on what Splunk server (not the host, source, or sourcetype) sent the logs.

For example:

Splunk servers A, B and C forward their logs to Splunk server D. If Splunk server D received data from Splunk server C, Splunk server D must index+forward the data to Splunk server E (regardless of the event details). Splunk server D will only index, and not forward, logs forwarded from Splunk servers A and B.

Please note: Clustering is not an option in our environment

Thank you for any feedback

0 Karma

FrankVl
Ultra Champion

I assume you are aware of the generic configuration options for routing data and such? http://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad

Basically, you will need something in the events that identifies the forwarder it came from. If you happen to have that (e.g. source value contains a file path that includes a forwarder specific piece), you could use that for the filtering, but I would expect you probably have some data sources where you do not have such identifying info in the event?

One solution you could look at is configure multiple splunktcp inputs (separate ports) on forwarder D, each with its own related forwarding and indexing config and then send A and B to one input and C to the other. But while that is a nice idea with a handful of forwarders like your example, this could become a bit complicated if the real situation is much more complex.

Alternatively you may just have to rethink your architecture and not send data from C to D in the first place, but get it to E directly or through another intermediate forwarder F.

0 Karma

bryanrobertson
New Member

Thank you. These are all good ideas. I have read over the “Route and filter data” and I use props.conf and transforms.conf in other places for routing purposes. The event data is so different in some cases that there are no easily repeatable patterns to make routing decisions. In the long term I am going to make the data easier to identify (separate indexes or create custom fields at index time), but all of that would require a lot of work for our setup.
I am going to try the multiple splunktcp inputs next week as that sounds like the easiest for our environment.

0 Karma

FrankVl
Ultra Champion

Alright, good luck! Looking forward to hear if you succeeded 🙂

0 Karma

David_Naylor
Path Finder

Hi Bryanrobertson,

Yes you can specify _INDEX_AND_FORWARD_ROUTING or _TCP_ROUTING under the following stanza

See the inputs.conf doc for more details on that stanza

Hope this helps, goodluck! - David

0 Karma

FrankVl
Ultra Champion

How exactly will that help his need to make routing decisions based on which upstream forwarder was sending?

0 Karma

David_Naylor
Path Finder

My theory was, that on the intermediate forwarder's input.conf you can specify the following stanza

  • Forwarder-specific settings for splunktcp.

  • Receivers use this input stanza.

  • This is the same as the [tcp://] stanza, except the remote server is assumed
    to be a Splunk instance, most likely a forwarder.

Under that stanza they should be able to specify forwarder specific settings for _INDEX_AND_FORWARD_ROUTING or _TCP_ROUTING parameters to route the events on a per forwarder basis.

0 Karma

bryanrobertson
New Member

Thank you. After reading over the inputs.conf specs file more, I think grouping the forwarders under the splunktcp stanza and using TCP_ROUTING will work for our case. I will give it a go next week

0 Karma

FrankVl
Ultra Champion

Unless you suggest to create separate splunktcp inputs for each (group of) forwarder(s), my question is: how to "specify forwarder specific settings for _INDEX_AND_FORWARD_ROUTING or _TCP_ROUTING parameters"?

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...