Ideal Heavy Forwarder to Universal Forwarder ratio?

[ishaanshekhar]

Dear SPLUNK Community,

I have around 150 UF, 2 HF and 4 Indexers.

I intend to auto load balance data from UF->HF and also HF->Indexer.

Is it fine to connect 150 UFs to 2 HFs? Do I need to change any configuration- ex. thruput parameter (in limits.conf) or maxQueueSize (in outputs.conf) in the UF/HF/Indexer?

Thanks in advance!
Ishaan

[martin_mueller · Accepted solution]

The heavy forwarders will be fine.

However, your two heavy forwarders will only firehose to up to two indexers at a time. As a result, your indexing load balancing is skewed and - in a worst-case scenario - will equal a rolling denial of service on your indexers.
On the other hand, the remaining two or three indexers will have nothing to index while the heavy forwarders are sending to the other indexer(s). That'll also skew search load balancing more than necessary later on.

If you absolutely want to add this extra forwarding tier, make sure you have more heavy forwarders than indexers to mitigate this.

From a config perspective, the universal forwarders don't need any changes. They just forward their data, and don't really care to whom they forward.
The heavy forwarders don't have a thruput limit by default, so you don't need to increase that. (Sidenote: this contributes to the rolling denial of service - a heavy forwarder can often do more thruput than an indexer can index.)
The indexers also don't need config changes, they don't really care where their data is coming from. You can't fix the rolling DoS on the config side anyway...

[martin_mueller]

All you really need is the server= setting. Leaving everything else unset (= defaults) is fine.

[martin_mueller]

Indexers can filter before the license meter just as well.

You need a heavy forwarder if you want to filter at the source, before ever sending data over the wire. For example, if your network between the machines with your UFs and the indexers is slow then you may want to use HFs instead of the UFs and filter before sending. Similarly, if you're filtering out data that legally may not be transferred off that machine you'll want HFs instead of UFs too.

[ishaanshekhar]

Thanks a ton! @martin_mueller . Really appreciate your expertise knowledge...

I may not use HF in this scenario. That leaves me with my last question on this topic....

On UFs my outputs.conf is like:

[tcpout:]
server = Ind_1:port, Ind_2:port, Ind_3:port, Ind_4:port
autoLB = true
autoLBFrequency = 20

Should I also add this (or any other parameter?) to avoid any DoS or any other issue?
forceTimebasedAutoLB = true

Thanks!!!
Ishaan

[martin_mueller]

Splitting things up artificially like that doesn't really change things, just makes your environment more complicated.

Let your UFs load balance directly to your indexers.

[ishaanshekhar]

Thank you, @martin_mueller . I read somewhere that we can filter events off license meter only at Heavy Forwarder level. Can it be done at Indexer level as well?

[martin_mueller]

forceTimebasedAutoLB won't change much to the DoS aspect.
useAck is good if you have an indexer cluster and want to make sure 100.000% of events get through even during a failure.

[ishaanshekhar]

Thanks Martin! I am sorry but I am not that knowledgeable on this DoS subject. Could you please suggest me some articles that I could go through?

[ishaanshekhar]

Could you please also suggest if I could route the data differently from HFs to Indexers?

e.g. HF1 -> Ind1, Ind2
and HF2 -> Ind3, Ind4

Or anything else?

Thanks a ton, @martin_mueller

[MuS]

+1 on the .conf and the DOS on the indexers

[ishaanshekhar]

Thank you Martin and MuS.

I read more on this topic. Would adding the following line in outputs.conf in UFs, and HFs solve this issue?
forceTimebasedAutoLB = true

Additionally, should I also consider useAck = true on HF side?