I am trying to do this:
Universal Forwarder1--> TCP 9997 --> Universal Forwarder2--> TCP 9997 --> Indexer (Search Head)
I see packets from UF 1 leaving on firewalls and received on UF 2, but not sure which file i need to make UF 2 take those packets and send to Indexer
Is this Input.conf at UF 2 that i need to modify? How does UF 2 know that it needs to take packetes from UF 1 and fwd to Indexer.
In order to enable the intermediary forwarders to listen to data, you have to add a stanza to your inputs.conf
[splunktcp://9997]
Or, if you want SSL:
[splunktcp-ssl://9997]
A universal forwarder does no local indexing, so the events will be forwarded, as specified in outputs.conf, by default. You can specify which indexes should be forwarded by changing the blacklists and whitelists in outputs.conf. Have a look at system/default/outputs.conf to see what is blacklisted and whitelisted by default. Basically, everything is forwarded, except anything starting with an underscore.
Two comments in general:
On the intermediate forwarder (UF2) you'll need to enable receiving and forwarding. From the CLI, you'd do this:
$SPLUNK_HOME/bin/splunk add forward-server yourserver:9997
(possibly multiple more receivers)
$SPLUNK_HOME/bin/splunk enable listen 9997
The first line modifies outputs.conf by adding the servers to the default group, the last line modifies inputs.conf by adding a [splunktcp://9997]
stanza.
Note, I've seen _internal
indexes sent from UFs to indexers but dropped when putting an intermediate forwarder in between. If you see that as well and want to forward them on, add this to the [tcpout]
stanza of outputs.conf on the intermediate forwarder (UF2):
forwardedindex.0.whitelist = .*
forwardedindex.1.whitelist = (_internal|_audit)
forwardedindex.2.blacklist = _.*