How to implement load balance in UF, HF, DS & IDX?

[Jagadeesh2022]

Hi Friends,

In our project, We have 3 regions. Each region has One HF and 30+ UF.  Same region UF's servers connected with corresponding  HF.  All the 3 region UF’s connected with DS.

Recently we faced below issue: 

One of the region HF server went down. So that region UF's data not received into Indexer.  

To overcome this issue in future how to implement load balancing in our project. I'm new to Splunk development . I want to know where to implement LB and how to implement?

Please guide me how to achieve this. Thanks in advance. 

[Jagadeesh2022]

Hi Friends,

My need is:  Whenever my configured HF went down it needs to apply failover and another HF needs to receive the data from UF's.  How to achieve this? Kindly share what I need to mention in output.cong file in UF server. 

It should not failover the HF after certain time period or some amount of data received.  It should failover only current HF went down. 

Thanks in advance for your suggestions. 

[richgalloway]

Splunk recommends having at least 2 intermediate forwarders for this very reason.  Load balancing is trivial in this use case and doesn't involve any network LBs.

1 - Stand up a new HF in each region

2 - On each UF, change the server= line in outputs.conf to add the new HF's name/address.

[tcpout:my_IFs]
server = HF_one:9997, HF_two:9997

3 - Restart the UFs

The UFs will automatically cycle through all of the addresses on the server= line.

[Jagadeesh2022]

Hi @richgalloway 

Thanks for your suggestion. 

I want to understand clearly  like if we mention 2 different HF names in each UF server. 

Which situation UF will send data to which HF? Example. I have configured HF1 & HF2 in all UF.

If  HF1 went down then it route to HF2 to send data or both HF's active, UF will send data first 1 hour to HF1 then it will send to HF2.  How it will work? 

Thanks in advance for your response. 

[richgalloway]

The UFs will send to HF1 for 30 seconds and then to HF2 for 30 seconds, then back to HF1 and so on.  The interval is set by autoLBFrequency in outputs.conf.  Of course, if one HF is down, the UF will continue to use the one that is available.

[Jagadeesh2022]

Hi @richgalloway 

My need is:  Whenever my configured HF1 went down it needs to apply failover and another HF2 needs to receive the data from UF's.  How to achieve this? Kindly share what I need to mention in output.cong file in UF server. 

It should not failover the HF after certain time period or some amount of data received.  It should failover only current HF1 went down. 

Thanks in advance for your suggestions.

[richgalloway]

I'm not sure that can be done.  Splunk auto-load balances by default.  There are options that govern how the load balancing behaves, but none that disable it or turn it into a failover-only feature.