Getting Data In

How to configure inputs.conf and outputs.conf on the Heavy Forwarder to route data received from universal forwarders to the indexers?

Olamide22
Explorer

Hello All -

We currently have a distributed architecture that's laid out in the following manner : UF ---> Indexers ---> SH

We now want to put in a heavy forwarder between the UF and the Indexers, i.e; UF ---> HF ---> Indexers ---> SH

The new architecture will enable us to perform parsing on the HF instance, as well as forward data to 3rd parties. Local indexing will be disabled on the HFs.

Since we are receiving data from UF(on multiples servers) on various events sources with different sourcetypes and are currently being indexed in different indexes, what do I need to configure in the inputs.conf and outputs.conf of my heavy forwarders ? The outputs.conf on the HF will be configured to forward data to the indexers. Essentially, my question is what inputs and outputs config do I need to on my HF to make sure that the various data being sent over to my HFs from my UFs are forwarded to the indexes(on the Indexers) specified in my UFs inputs.conf

  • Will only enabling the HF to listen on TCP 9997 suffice for receiving the various data streams from the UF and the subsequent forwarding to the respective indexes?
  • Or do I need to selectively route data using _TCP_ROUTING = to get data to my desired index on the indexers?

The other option that I came across in the outputs.conf only described routing to syslog server.

Thanks in advance for your responses.

1 Solution

jayannah
Builder

For reading from all UFs, enabling listen on 9997 will suffice.

                       Inputs.conf
                      [splunktcp:9997]

For outputs.conf to send it to indexers and aswell as to 3rd party (3rd party meaning non splunk instance..correct??)

                    outputs.conf
                    [tcpout]
                    defaultGroup = default-autolb-group , thridparty_group

                     #Splunk indexers
                    [tcpout:default-autolb-group]
                    server = idx1:9997,idx2:9997,idx3:9997
                    autoLB = true

                    #send to 3rd party (non splunk instances)
                    [tcpout:thridparty_group]
                    server = ip1:port, ip2:port
                    autoLB = true
                    sendCookedData = false

P.S: sendCookedData = false will send the raw events and untouched prior to sending

View solution in original post

jayannah
Builder

For reading from all UFs, enabling listen on 9997 will suffice.

                       Inputs.conf
                      [splunktcp:9997]

For outputs.conf to send it to indexers and aswell as to 3rd party (3rd party meaning non splunk instance..correct??)

                    outputs.conf
                    [tcpout]
                    defaultGroup = default-autolb-group , thridparty_group

                     #Splunk indexers
                    [tcpout:default-autolb-group]
                    server = idx1:9997,idx2:9997,idx3:9997
                    autoLB = true

                    #send to 3rd party (non splunk instances)
                    [tcpout:thridparty_group]
                    server = ip1:port, ip2:port
                    autoLB = true
                    sendCookedData = false

P.S: sendCookedData = false will send the raw events and untouched prior to sending

rmorlen
Splunk Employee
Splunk Employee

Outputs.conf would point to your indexers from the HF.

outputs.conf would point to the HF from your UF's.

We have out HF's listening on different ports for data. So inputs.conf would contain something like:

[tcp://9997]
sourcetype=http

[tcp://9192]
sourcetype=os

We also put any props and transforms on the HF's. We actually have separate props apps but one would do fine. We just have a lot of data feeds.

You might also look at: http://answers.splunk.com/answers/169929/using-heavy-forwarders-as-an-intermediary-layer.html

Olamide22
Explorer

Any takers?

0 Karma

jofe
Explorer

Heavy Forwarder should problably look something like this.

Inputs.conf
[splunktcp:9997]
connection_host = dns (or IP if you prefer that)

Outputs.conf
[tcpout:d1]
server=d1-splunkix-01:9997,d1-splunkix-02:9997 (if you have two indexers)
autoLB = true

If you have defined index, source and sourcetype on the UF (Splunk agent) then you don't need to define that on the heavy forwarder. It will just forward those fields as they were.

IMPORTANT : Remember that any INDEX-time configuration (line breaking, filtering etc.) must be moved out from indexers to your heavy forwarders. 🙂

Then change the destination where you UFs send their data to the Heavy forwarders.

0 Karma
Get Updates on the Splunk Community!

Splunk is Nurturing Tomorrow’s Cybersecurity Leaders Today

Meet Carol Wright. She leads the Splunk Academic Alliance program at Splunk. The Splunk Academic Alliance ...

Part 2: A Guide to Maximizing Splunk IT Service Intelligence

Welcome to the second segment of our guide. In Part 1, we covered the essentials of getting started with ITSI ...

Part 1: A Guide to Maximizing Splunk IT Service Intelligence

As modern IT environments continue to grow in complexity and speed, the ability to efficiently manage and ...