topic Re: Ideal Heavy Forwarder to Universal Forwarder ratio? in Deployment Architecture

Ideal Heavy Forwarder to Universal Forwarder ratio?

ishaanshekhar — Sun, 13 Sep 2015 09:22:17 GMT

Dear SPLUNK Community,

I have around 150 UF, 2 HF and 4 Indexers.

I intend to auto load balance data from UF->HF and also HF->Indexer.

Is it fine to connect 150 UFs to 2 HFs? Do I need to change any configuration- ex. thruput parameter (in limits.conf) or maxQueueSize (in outputs.conf) in the UF/HF/Indexer?

Thanks in advance!
Ishaan

Re: Ideal Heavy Forwarder to Universal Forwarder ratio?

martin_mueller — Sun, 13 Sep 2015 10:41:44 GMT

The heavy forwarders will be fine.

However, your two heavy forwarders will only firehose to up to two indexers at a time. As a result, your indexing load balancing is skewed and - in a worst-case scenario - will equal a rolling denial of service on your indexers.
On the other hand, the remaining two or three indexers will have nothing to index while the heavy forwarders are sending to the other indexer(s). That'll also skew search load balancing more than necessary later on.

If you absolutely want to add this extra forwarding tier, make sure you have more heavy forwarders than indexers to mitigate this.

From a config perspective, the universal forwarders don't need any changes. They just forward their data, and don't really care to whom they forward.
The heavy forwarders don't have a thruput limit by default, so you don't need to increase that. (Sidenote: this contributes to the rolling denial of service - a heavy forwarder can often do more thruput than an indexer can index.)
The indexers also don't need config changes, they don't really care where their data is coming from. You can't fix the rolling DoS on the config side anyway...

Re: Ideal Heavy Forwarder to Universal Forwarder ratio?

MuS — Sun, 13 Sep 2015 10:48:47 GMT

+1 on the .conf and the DOS on the indexers

Re: Ideal Heavy Forwarder to Universal Forwarder ratio?

ishaanshekhar — Sun, 13 Sep 2015 15:00:36 GMT

Thank you Martin and MuS.

I read more on this topic. Would adding the following line in outputs.conf in UFs, and HFs solve this issue?
forceTimebasedAutoLB = true

Additionally, should I also consider useAck = true on HF side?

Re: Ideal Heavy Forwarder to Universal Forwarder ratio?

martin_mueller — Sun, 13 Sep 2015 15:02:26 GMT

forceTimebasedAutoLB won't change much to the DoS aspect.
useAck is good if you have an indexer cluster and want to make sure 100.000% of events get through even during a failure.

Re: Ideal Heavy Forwarder to Universal Forwarder ratio?

ishaanshekhar — Sun, 13 Sep 2015 15:26:39 GMT

Thanks Martin! I am sorry but I am not that knowledgeable on this DoS subject. Could you please suggest me some articles that I could go through?

Re: Ideal Heavy Forwarder to Universal Forwarder ratio?

ishaanshekhar — Sun, 13 Sep 2015 16:13:13 GMT

Could you please also suggest if I could route the data differently from HFs to Indexers?

e.g. HF1 -> Ind1, Ind2
and HF2 -> Ind3, Ind4

Or anything else?

Thanks a ton, @martin_mueller

Re: Ideal Heavy Forwarder to Universal Forwarder ratio?

martin_mueller — Sun, 13 Sep 2015 16:14:47 GMT

Splitting things up artificially like that doesn't really change things, just makes your environment more complicated.

Let your UFs load balance directly to your indexers.

Re: Ideal Heavy Forwarder to Universal Forwarder ratio?

ishaanshekhar — Sun, 13 Sep 2015 18:11:58 GMT

Thank you, @martin_mueller . I read somewhere that we can filter events off license meter only at Heavy Forwarder level. Can it be done at Indexer level as well?

Re: Ideal Heavy Forwarder to Universal Forwarder ratio?

martin_mueller — Sun, 13 Sep 2015 18:45:54 GMT

Indexers can filter before the license meter just as well.

You need a heavy forwarder if you want to filter at the source, before ever sending data over the wire. For example, if your network between the machines with your UFs and the indexers is slow then you may want to use HFs instead of the UFs and filter before sending. Similarly, if you're filtering out data that legally may not be transferred off that machine you'll want HFs instead of UFs too.

Re: Ideal Heavy Forwarder to Universal Forwarder ratio?

ishaanshekhar — Tue, 29 Sep 2020 07:16:05 GMT

Thanks a ton! @martin_mueller . Really appreciate your expertise knowledge...

I may not use HF in this scenario. That leaves me with my last question on this topic....

On UFs my outputs.conf is like:

[tcpout:]
server = Ind_1:port, Ind_2:port, Ind_3:port, Ind_4:port
autoLB = true
autoLBFrequency = 20

Should I also add this (or any other parameter?) to avoid any DoS or any other issue?
forceTimebasedAutoLB = true

Thanks!!!
Ishaan

Re: Ideal Heavy Forwarder to Universal Forwarder ratio?

martin_mueller — Mon, 14 Sep 2015 22:23:39 GMT

All you really need is the server= setting. Leaving everything else unset (= defaults) is fine.