Getting Data In

Limitation of Heavy Forwarder outputs?

LuanNguyen
Engager

I wonder if a Heavy Forwarder can be the intermediate instance among 1000 Universal Forwarders and 1000 Indexers? The hardware resources are supposed to be unlimited, the problem will be only about the configuration.

Any documentations or references will be big helps. Thank you very much!

0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

Yes, a heavy forwarder can be used in that manner.  Having only one HF, however, is a single point of failure that could lead to data loss if it is unavailable.  Be sure to set up at least 2 intermediate forwarders.

See https://www.linkedin.com/pulse/splunk-asynchronous-forwarding-lightning-fast-data-ingestor-rawat/ for how to configure the HFs for better performance in this situation.

What specific questions do you have about the configuration?

---
If this reply helps you, Karma would be appreciated.

View solution in original post

gcusello
SplunkTrust
SplunkTrust

Hi @LuanNguyen ,

yes, yupu can use an HF as intermediate Forwarder between UFs and IDXs.

The number is relevant only to correctly have a dimensioning of the reference hardware.

At first I hint to engage a Splunk Architect for this job.

Then I hint to avoid a single point of failure using at least two or three HFs.

Then there isn't a reference hardware for the HF, in my experience we started with the default HW reference (12 CPUs, 12 GB RAM, 300 GB disk), and then, analyzing the use of these resources, we defined to add some CPUs.

In addition, you should define if these HFs are only  concentrators or if they also do parsing, merging and typing phases, especially the parsing phase: many transformations requires more resources.

Then, if you have many UFs, you could prefer to have three or four HFs instead of two with more resources, to avoid that the network interfaces are the bottleneck.

As I said, this design requires at least a Splunk Architect or a Splunk PS.

Ciao.

Giuseppe 

LuanNguyen
Engager

Thanks for your practical answer, this is not what I asked for but really what I need. Appreciate it very much!

richgalloway
SplunkTrust
SplunkTrust

Yes, a heavy forwarder can be used in that manner.  Having only one HF, however, is a single point of failure that could lead to data loss if it is unavailable.  Be sure to set up at least 2 intermediate forwarders.

See https://www.linkedin.com/pulse/splunk-asynchronous-forwarding-lightning-fast-data-ingestor-rawat/ for how to configure the HFs for better performance in this situation.

What specific questions do you have about the configuration?

---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

Database Performance Sidebar Panel Now on APM Database Query Performance & Service ...

We’ve streamlined the troubleshooting experience for database-related service issues by adding a database ...

IM Landing Page Filter - Now Available

We’ve added the capability for you to filter across the summary details on the main Infrastructure Monitoring ...

Dynamic Links from Alerts to IM Navigators - New in Observability Cloud

Splunk continues to improve the troubleshooting experience in Observability Cloud with this latest enhancement ...