How to forward logs from universal forwarders to heavy forwarders for filtering before routing to indexers?

[thezero]

Hi Guies,

We have multiple universal forwarders and 3 heavy weight forwarders. Currently all UFs are forwarding logs directly to indexers. What I want is to configure universal forwarder to send logs/data to heavy weight forwarders and do some filtering there, and then forward the logs to indexers from heavy weight forwarders. How can I achieve this? What additional config do I need to do on the heavy weight forwarders?

What I want to achieve is:

Universal forwarder ----->Heavy weight forwarder ----->Indexer

• for forwarding data from UF to heavy weight forwarder, I will edit outputs.conf on universal forwarder
• what config should I do on the heavy weight forwarders to collect data from universal forwarders and route it to the indexers?
• do I need to configure receiving on the HWF?

Please advise

[yomesky2000]

Thanks for the explanation. However, I will like to know if it is ideal for the Heavy Forwarder to be forwarding to the Indexer Master?

I understand that the Indexer master is not part of the Indexer cluster and it only control the activities on the IDX cluster, but i am just just curious how do it do that.

My question:
1. Can the HF ----- IDX MASTER ? Can the HF forward data to the IDX Master too?
2. Can the Indexer Master be added to the RECIEVING PEERS like the Indexers?
3. If we have 2 HF and we want to forwarder data from UF to the HF, do we have to run the command to forward logs to the HF:9997 twice to achieve that?
4. Is there any way to cluster Heavy Forwarder in SplunK

Please advice

[MuS]

Hi yomesky2000,

I will split the answer to your 4 questions into two answers:

1+2: Technically you can; should you do that? No! Your IDXC Master will be busy doing things its supposed to do and you will get issues as soon as your IDXC recovers for example. Also it is not recommended to do so, see the docs https://docs.splunk.com/Documentation/Splunk/latest/Deploy/Manageyourdeployment#Whether_to_colocate_...

3+4: Use a deployment server to configure the 2 HWF's so they use the same config files and enable forwarding using an outputs.conf config file, see the docs https://docs.splunk.com/Documentation/Splunk/latest/Admin/Outputsconf#outputs.conf.example
No, there is no clustering for HWF but using a deployment server and configuring a DNS round-robin CNAME that uses the IP's of both HWF's you get as close as possible, next you point your UF's to send to this CNAME. Don't use a physical load balancer in front of HWF for reasons, UF's can handle load balancing themselves and also the error handling of connections - which will a load balancer break if you put one in the Splunk-2-Splunk traffic.

Hope this helps ...

cheers, MuS

[markusspitzli]

First of all you want to enable the Indexing port on your Heavyforwarder, otherwise you wont be able to get the data.

[splunktcp-ssl://9997]

Then you have to configure the outputs.conf. someting like this:

[tcpout]
defaultGroup = mygroup

[tcpout:mygroup]
server = indexer1:9997,indexer2:9997

And the UF should now point to your Heavyforwarder in its outputs.conf
[tcpout]
defaultGroup = mygroup

[tcpout:mygroup]
server = heavyforwarder:9997

Keep in mind that your Heavyforwarder is now responsible for the parsing phase. Put every TA on it for extracting timestamps and do the linebreaking correctly. You want to do both manually for performance reasons. someting like this:

[mysourcertype]
SHOULD_LINEMERGE = false
TRUNCATE = 100000
MAX_TIMESTAMP_LOOKAHEAD = 30
NO_BINARY_CHECK = true
TIME_FORMAT=%F %T,%3N
TIME_PREFIX=^
LINE_BREAKER=([\r\n]+)\d{4}\-\d{2}\-\d{2}

[DBattisto]

Amazing that you reply to a thread several years old the week I start working on a similar issue. Perhaps you could provide some feedback to my plan? 🙂

Windows PC w/ Universal Forwarder --------> Heavy Forwarder ---------> Indexer

Windows PC:
input - Collecting log inputs through Universal Forwarder inputs.conf
output - outputs.conf to heavyfwdr xx.xx.xx.xx:9997

Heavy Forwarder:
input - Windows TA inputs.conf listening on port 9997?
output - outputs.conf to indexer:9997

Question:
1. Is this the heavy forwarder the best place for Windows TA, or should it be on the Universal Forwarders?
2. If it is acceptable as-is, how do I configure the heavy forwarder to listen on 9997 and direct it to the Windows TA? Is that done in the Windows TA inputs.conf?

[markusspitzli]

Hey

The config is the same as described above.
You dont have to config anything regarding your Windows TA. You just have to copy on the HF and it works.

if you the whole Windows-TA 1:1 on the Heavyforwarder, be aware that only two configfiles are needed ( props,conf and transforms.conf)

In our environment I make a new App (like Windows-TA-hf, depends on your naming convention) and put only the needed files into this app.
We manually define Linebreaking and the Timestamp because of Splunks Best Practice and of performance reason.

[DBattisto]

In the event that someone else has this issue and sees this thread, I created an environment where:

Universal Forwarder --> Heavy Forwarder --> Indexer

I had to draw out the setup on paper to visualize what files needed to be configured, and which ports/firewall rules needed to be opened/modified. Thanks for the help!

[MuS · Accepted solution]

Hi thezero,

Take a look at the docs http://docs.splunk.com/Documentation/Splunk/6.3.1/Forwarding/Deployaheavyforwarder which will guide you through the process of enabling a heavy weight forwarder.

Hope this helps ...

cheers, MuS

[thezero]

Hi MuS,

Thanks for your suggestion.I gone through the doc but it does not answer my query. my confusion is as below?

1.How can we forward logs to heavy weight forwarder from universal forwarder --->configuring outputs.conf in universal forward to forward lopgs to heavy weight forwader will do the task .I assume.Is that right?

2.for receiving data from universal forwarder do I need to enable receiving in heavy weight forwader?

3.configuring forwarding in heavy weight forwader will automatically forward data received from universal forwarder or do I need to do some additional config?

please advise

[MuS]

1. yes
2. yes
3. yes and no; just make sure that in the outputs.conf you set indexAndForward = false otherwise you will index the forwarded events from the universal forwarder as well on the heavy weight forwarder.

Hope this helps ...

[thezero]

Hi Mus,

That worked like charm.thx 🙂

[ajitshukla]

I am also facing same problem here is my problem statement:

1.My universal forwarder sending Binary data to Heavy Forwarder in Index name as "Binary_index" .
2.On heavy Forwarder I want to convert these Binary data to csv format,for which I have written python script and then send CSV data to splunk instance.

But I don't know I to achieve this
please provide me solution with configuration files details for every steps.

It would be great help if you provide detailed solution for this,since I am new to splunk .

please provide the configuration files details for reading binary data on universal forwarder side.