Splunk Heavy Forwarder vs Universal Forwarder for large splunk instance

[Twagner79]

Hello Everyone,  looking for a little guidance on our Splunk deployment for a system.  Currently, we have a few different sites that span across the US with Universal forwarders deployed to all of the systems and reporting back to one main Splunk instance individually.  

Id like to see about splitting the Splunk system up into two separate parts to improve integrity and reduce latency, but have never delt with deploying a heavy forwarder in an instance like this.  My thought is to have all of the western universal forwarders sending their events to a dedicated Western Heavy forwarder, and have all of the eastern universal forwards send their data to the eastern heavy forwarder, and have both of the heavy forwarders send their data to our main Splunk instance.  (crude visio below) Any guidance is greatly appreciated! 

 

[Twagner79]

Shooting out one Response since it applies to all. Thank you everyone for your guidance and knowledge! Always helps to have a strong community to back some of these questions so I appreciate it!

[richgalloway]

If your problem is resolved, then please click an "Accept as Solution" button to help future readers.

[PickleRick]

There are general recommendations in form of so called SVAs - Splunk Validated Architectures

https://docs.splunk.com/Documentation/SVA/current/Architectures/About

Of course those are guidelines and are based on typical needs and can be adjusted in some border cases - for example when you have a relatively isolated environment you can (and often will) use intermediate forwarder(s) to route events from sources inside to the indexers on the outside. But the intermediate forwarder can (and often will) be a UF - it doesn't have to be a HF.

While I like the (non-SVA) general concept of additional HF layer in front of indexers (it has its pros and cons so it's not a solution I'll preach indiscriminently to everyone), adding multiple HFs in a single "branched" processing chain introduces a whole lot of inconsistency and can be a huge pain to troubleshoot in case of problems with ingestion.

[gcusello]

Hi @Twagner79 ,

as also @richgalloway and @isoutamo said, there's no utility to use an intermediate Forwarder as concentrator, it's better to directly send logs to the Indexers.

The only application I know (and I applied) of this solution is when you have forwarders in a restricted network and you don't want to open many firewall routes between all the forwarders and the Indexers, but it shouldn't be your case.

In addition, adding an additional layer doesn't reduce latency but increases it and at the same time doesn't give any improvement to integrity, that's guaranteed by the use of Forwarders that have a local cache.

Ciao.

Giuseppe

[richgalloway]

Adding intermediate forwarders introduces complexity and improves neither integrity nor latency. 

Loss of one of the HFs means half of the UFs are off-line.

The HFs need time to process events so that adds latency.

[isoutamo]

Hi

The splunk best practices is use UFs and sends logs directly into indexers. Of course there are some cases where it's best to use also HFs between UF and indexers, but not in normal case.

When you adding HF between UF and indexer you always add complexity, latency in your installation. Also in most cases you also reduce event distribution on indexer sides which decrease your search performance.

Using HFs instead of UF will also generate more traffic between sites as HF's add some metadata on all events.

Based on what you have told, I don't see that this separation will lead to your objectives, instead it do just opposite result.

But if you still want to do it, then you should change at least next part

• connect those HF's directly to your main splunk, not West -> East -> Indexers
• add more HFs on both site to get redundancy and better event distribution and performance
• add more pipelines in every HFs to get better performance and event distribution on indexers

r. Ismo