topic Re: Limitation of Heavy Forwarder outputs? in Getting Data In

Limitation of Heavy Forwarder outputs?

LuanNguyen — Mon, 15 Apr 2024 11:19:40 GMT

I wonder if a Heavy Forwarder can be the intermediate instance among 1000 Universal Forwarders and 1000 Indexers? The hardware resources are supposed to be unlimited, the problem will be only about the configuration.

Any documentations or references will be big helps. Thank you very much!

Re: Limitation of Heavy Forwarder outputs?

richgalloway — Mon, 15 Apr 2024 11:58:34 GMT

Yes, a heavy forwarder can be used in that manner. Having only one HF, however, is a single point of failure that could lead to data loss if it is unavailable. Be sure to set up at least 2 intermediate forwarders.

See https://www.linkedin.com/pulse/splunk-asynchronous-forwarding-lightning-fast-data-ingestor-rawat/ for how to configure the HFs for better performance in this situation.

What specific questions do you have about the configuration?

Re: Limitation of Heavy Forwarder outputs?

gcusello — Mon, 15 Apr 2024 12:03:47 GMT

Hi @LuanNguyen ,

yes, yupu can use an HF as intermediate Forwarder between UFs and IDXs.

The number is relevant only to correctly have a dimensioning of the reference hardware.

At first I hint to engage a Splunk Architect for this job.

Then I hint to avoid a single point of failure using at least two or three HFs.

Then there isn't a reference hardware for the HF, in my experience we started with the default HW reference (12 CPUs, 12 GB RAM, 300 GB disk), and then, analyzing the use of these resources, we defined to add some CPUs.

In addition, you should define if these HFs are only concentrators or if they also do parsing, merging and typing phases, especially the parsing phase: many transformations requires more resources.

Then, if you have many UFs, you could prefer to have three or four HFs instead of two with more resources, to avoid that the network interfaces are the bottleneck.

As I said, this design requires at least a Splunk Architect or a Splunk PS.

Ciao.

Giuseppe

Re: Limitation of Heavy Forwarder outputs?

LuanNguyen — Tue, 16 Apr 2024 02:25:54 GMT

Thanks for your practical answer, this is not what I asked for but really what I need. Appreciate it very much!