How to configure inputs.conf and outputs.conf on the Heavy Forwarder to route data received from universal forwarders to the indexers?

[Olamide22]

Hello All -

We currently have a distributed architecture that's laid out in the following manner : UF ---> Indexers ---> SH

We now want to put in a heavy forwarder between the UF and the Indexers, i.e; UF ---> HF ---> Indexers ---> SH

The new architecture will enable us to perform parsing on the HF instance, as well as forward data to 3rd parties. Local indexing will be disabled on the HFs.

Since we are receiving data from UF(on multiples servers) on various events sources with different sourcetypes and are currently being indexed in different indexes, what do I need to configure in the inputs.conf and outputs.conf of my heavy forwarders ? The outputs.conf on the HF will be configured to forward data to the indexers. Essentially, my question is what inputs and outputs config do I need to on my HF to make sure that the various data being sent over to my HFs from my UFs are forwarded to the indexes(on the Indexers) specified in my UFs inputs.conf

• Will only enabling the HF to listen on TCP 9997 suffice for receiving the various data streams from the UF and the subsequent forwarding to the respective indexes?
• Or do I need to selectively route data using _TCP_ROUTING = to get data to my desired index on the indexers?

The other option that I came across in the outputs.conf only described routing to syslog server.

Thanks in advance for your responses.

[jayannah · Accepted solution]

For reading from all UFs, enabling listen on 9997 will suffice.

                       Inputs.conf
                      [splunktcp:9997]

For outputs.conf to send it to indexers and aswell as to 3rd party (3rd party meaning non splunk instance..correct??)

                    outputs.conf
                    [tcpout]
                    defaultGroup = default-autolb-group , thridparty_group

                     #Splunk indexers
                    [tcpout:default-autolb-group]
                    server = idx1:9997,idx2:9997,idx3:9997
                    autoLB = true

                    #send to 3rd party (non splunk instances)
                    [tcpout:thridparty_group]
                    server = ip1:port, ip2:port
                    autoLB = true
                    sendCookedData = false

P.S: sendCookedData = false will send the raw events and untouched prior to sending

[somesoni2]

Reading from documentation

http://docs.splunk.com/Documentation/Splunk/6.2.1/Forwarding/Configureforwarderswithoutputs.confd

http://docs.splunk.com/Documentation/Splunk/6.2.1/Forwarding/Setupforwardingandreceiving

[rmorlen]

Outputs.conf would point to your indexers from the HF.

outputs.conf would point to the HF from your UF's.

We have out HF's listening on different ports for data. So inputs.conf would contain something like:

[tcp://9997]
sourcetype=http

[tcp://9192]
sourcetype=os

We also put any props and transforms on the HF's. We actually have separate props apps but one would do fine. We just have a lot of data feeds.

You might also look at: http://answers.splunk.com/answers/169929/using-heavy-forwarders-as-an-intermediary-layer.html

[Olamide22]

Any takers?

[jofe]

Heavy Forwarder should problably look something like this.

Inputs.conf
[splunktcp:9997]
connection_host = dns (or IP if you prefer that)

Outputs.conf
[tcpout:d1]
server=d1-splunkix-01:9997,d1-splunkix-02:9997 (if you have two indexers)
autoLB = true

If you have defined index, source and sourcetype on the UF (Splunk agent) then you don't need to define that on the heavy forwarder. It will just forward those fields as they were.

IMPORTANT : Remember that any INDEX-time configuration (line breaking, filtering etc.) must be moved out from indexers to your heavy forwarders. 🙂

Then change the destination where you UFs send their data to the Heavy forwarders.