Is multitiered load balancing supported in Splunk 6.3.1? (Universal Forwarders > Heavy Forwarders > Clustered Indexers)

[javiergn]

Hi,

After going through the 6.3.1 documentation, it is still not clear to me whether multitiered load balancing is fully supported in Splunk. I don't see why not, but I just want to double check with the community.

This is the scenario I'm thinking about:

• 100 Universal Forwarders (read logs) -> 4 Heavy Forwarders (parse and obfuscate data) -> 2 Indexers (indexing and storage)
• UFs send data to 4 HFs using load balancing . Up to 3 HFs can be down any time
• HFs send parsed data to 2 IDXs using load balancing . Up to 1 IX can be down any time
• IDXs replicate and sync with each other so that data is kept in two different places

I just want to make sure there's no single point of failure here.

Thanks,
J

[mhassan]

Simple answer, yes it works. As for Indexer discovery question, this has to do with the HF (in your case) discovering any new IDX added to the cluster. The UF or HF are no longer required to be configured (outputs.conf) ahead of time. This feature only works for clustered indexers.

[ltrand · Accepted solution]

The UF's can send to a network LB to send to the HF's or can auto loadbalance on their own through outputs.conf. There are pro's and con's for each, Splunk's recommendation is against network load balancers.

[javiergn]

Yeah I would definitely prefer the autoLB option but in that case, would autoLB work in both UFs and HFs?
For instance, is the following possible?

####################
# outputs.conf UF1 - UF100
[tcpout]
server = HF1:9997, HF2:9997, HF3:9997, HF4:9997
autoLB = true
autoLBFrequency = 30

###################
# outputs.conf HF1 - HF4
[tcpout]
server = IX1:9997, IX2:9997
autoLB = true
autoLBFrequency = 30

[mikelanghorst]

Yes, this would work

[asimagu]

I am guessing that your Indexers will be clustered, right? you are talking about syncronizing a copy.....but I hope you are not just using "index and forward"

So,if you are thinking about clustering the IXs, the load balancing for the HWFs will now be managed by the cluster master in a smart way (the feature so called IndexerDiscovery).

There is also a new feature that you may want to enable if the total disk available is different in each node.

I hope that helped

[javiergn]

Hi,

Yeah, indexers will be clustered. In fact the scenario I'm talking about above is just a simplified version where only one site is required. In reality we are going to have multiple sites.

What do you mean by "the load balancing for the HWFs will now be managed by the cluster master"? My HFs are not doing any indexing and they won't be searchable from the Search Heads. They are not acting as Search Peers basically, they are just intermediate forwarders, so I'm not sure why you would want your cluster master to manage that. Unless I'm missing something here.

All I want to know is whether a UF can forward to multiple intermediate forwarders (HFs in this case) using load balancing and these then can forward to multiple indexers using load balancing too.

Thanks,
J