forwarding to multiple servers

[marksheinbaum]

We have a requirement to forward different data to multiple Splunk instances. In this case, security data is forwarded to EntServer2, where app data if forwarded to EntServer1. What is best practice regarding Universal Forwarders? Set up two universal forwarders on same app server, or, set up and configure a single UF to forward to both Ent Servers?

[PickleRick]

No. It could be complicated to install two UF instances on one host. Especially on Windows.

If you're configuring tcpout outputs, you can just set up two output groups and send to both.

[marksheinbaum]

Thanks for the reply. Regarding forwarding to two output groups, is there some doc that describes this in detail? In any case, given that we have multiple app log files and script output already going to one of the enterprise servers, how difficult will it be to segregate which log files go to which Enterprise server?

[PickleRick]

That's getting more complicated. Simple forwarding of all data to multiple groups is... simple. You define two or more output groups and send everything everywhere.

If you want to send all data from specific input to a given output(s), you can use the _TCP_ROUTE setting in input definition (see spec for inputs.conf).

But if you want to send only selected events to specific destinations, you're left with https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad It needs a HF - UF doesn't do parsing.

[marksheinbaum]

I read thru the Splunk docs and it seems like a UF with customized inputs.conf and output.conf file should work. If the two enterprise servers are defined in the output.conf file, than we can use the inputs conf stanza to customize the destination where various log files are sent. Just wanted to confirm before accepting this solution.

_TCP_ROUTING = <comma-separated list>
* A comma-separated list of tcpout group names.
* This setting lets you selectively forward data to one or more specific indexers.
* Specify the tcpout group that the forwarder uses when forwarding the data.
  The tcpout group names are defined in outputs.conf with
  [tcpout:<tcpout_group_name>].
* To forward data to all tcpout group names that have been defined in
  outputs.conf, set to '*' (asterisk).
* To forward data from the "_internal" index, you must explicitly set
  '_TCP_ROUTING' to either "*" or a specific splunktcp target group.
* Default: The groups specified in 'defaultGroup' in [tcpout] stanza in
  the outputs.conf file

[PickleRick · Accepted solution]

Right _TCP_ROUTING, not _TCP_ROUTE - my typo. But again - you can't selectively forward some part of data from a particular input to a specific output. It's all or nothing. If you can live witn it that's... something I'd still test in lab before pushing to prod. 😉