Getting Data In

How to configure inputs.conf and outputs.conf on the Heavy Forwarder to route data received from universal forwarders to the indexers?

Olamide22
Explorer

Hello All -

We currently have a distributed architecture that's laid out in the following manner : UF ---> Indexers ---> SH

We now want to put in a heavy forwarder between the UF and the Indexers, i.e; UF ---> HF ---> Indexers ---> SH

The new architecture will enable us to perform parsing on the HF instance, as well as forward data to 3rd parties. Local indexing will be disabled on the HFs.

Since we are receiving data from UF(on multiples servers) on various events sources with different sourcetypes and are currently being indexed in different indexes, what do I need to configure in the inputs.conf and outputs.conf of my heavy forwarders ? The outputs.conf on the HF will be configured to forward data to the indexers. Essentially, my question is what inputs and outputs config do I need to on my HF to make sure that the various data being sent over to my HFs from my UFs are forwarded to the indexes(on the Indexers) specified in my UFs inputs.conf

  • Will only enabling the HF to listen on TCP 9997 suffice for receiving the various data streams from the UF and the subsequent forwarding to the respective indexes?
  • Or do I need to selectively route data using _TCP_ROUTING = to get data to my desired index on the indexers?

The other option that I came across in the outputs.conf only described routing to syslog server.

Thanks in advance for your responses.

1 Solution

jayannah
Builder

For reading from all UFs, enabling listen on 9997 will suffice.

                       Inputs.conf
                      [splunktcp:9997]

For outputs.conf to send it to indexers and aswell as to 3rd party (3rd party meaning non splunk instance..correct??)

                    outputs.conf
                    [tcpout]
                    defaultGroup = default-autolb-group , thridparty_group

                     #Splunk indexers
                    [tcpout:default-autolb-group]
                    server = idx1:9997,idx2:9997,idx3:9997
                    autoLB = true

                    #send to 3rd party (non splunk instances)
                    [tcpout:thridparty_group]
                    server = ip1:port, ip2:port
                    autoLB = true
                    sendCookedData = false

P.S: sendCookedData = false will send the raw events and untouched prior to sending

View solution in original post

jayannah
Builder

For reading from all UFs, enabling listen on 9997 will suffice.

                       Inputs.conf
                      [splunktcp:9997]

For outputs.conf to send it to indexers and aswell as to 3rd party (3rd party meaning non splunk instance..correct??)

                    outputs.conf
                    [tcpout]
                    defaultGroup = default-autolb-group , thridparty_group

                     #Splunk indexers
                    [tcpout:default-autolb-group]
                    server = idx1:9997,idx2:9997,idx3:9997
                    autoLB = true

                    #send to 3rd party (non splunk instances)
                    [tcpout:thridparty_group]
                    server = ip1:port, ip2:port
                    autoLB = true
                    sendCookedData = false

P.S: sendCookedData = false will send the raw events and untouched prior to sending

rmorlen
Splunk Employee
Splunk Employee

Outputs.conf would point to your indexers from the HF.

outputs.conf would point to the HF from your UF's.

We have out HF's listening on different ports for data. So inputs.conf would contain something like:

[tcp://9997]
sourcetype=http

[tcp://9192]
sourcetype=os

We also put any props and transforms on the HF's. We actually have separate props apps but one would do fine. We just have a lot of data feeds.

You might also look at: http://answers.splunk.com/answers/169929/using-heavy-forwarders-as-an-intermediary-layer.html

Olamide22
Explorer

Any takers?

0 Karma

jofe
Explorer

Heavy Forwarder should problably look something like this.

Inputs.conf
[splunktcp:9997]
connection_host = dns (or IP if you prefer that)

Outputs.conf
[tcpout:d1]
server=d1-splunkix-01:9997,d1-splunkix-02:9997 (if you have two indexers)
autoLB = true

If you have defined index, source and sourcetype on the UF (Splunk agent) then you don't need to define that on the heavy forwarder. It will just forward those fields as they were.

IMPORTANT : Remember that any INDEX-time configuration (line breaking, filtering etc.) must be moved out from indexers to your heavy forwarders. 🙂

Then change the destination where you UFs send their data to the Heavy forwarders.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...