Load Balancing at UF to HF

[vr2312]

We have the current infrastructure :

UF -> HF -> Indexers

Can i set up Load Balancing at the outputs.conf so that data is forwarded equally to the HF ?

I would like to know the pros and cons for this.

[vr2312]

Thank you @harsmarvania57 and @FrankVI for your answers.

If i have one of my Server hosting 4 instances of HF, would i still be able to achieve this ?

[FrankVl]

Theoretically you could, by having those instances use different ports or bind them each to specific (virtual) ip addresses or something like that, but running multiple instances of Splunk on a single server is not supported by Splunk, so I wouldn't recommend doing that.

What was your intention with setting up 4 HFs on a single server?

[harsmarvania57]

If I am understanding your comment correctly, you are running single server with 4 different splunk instances running on same server and acting as HF, in that case you can achieve this because your all HF listening/receiving data from UF on different ports but I can't see any benefit for this one because if your server will go down then all 4 HF instances will go down and UF->HF data transfer will be stopped.

Any specific reason to run 4 different splunk instances on same server because it is not a good practice.

Please correct me if I misunderstood your comment.

[vr2312]

@harsmarvania57

Yes. You got me right.

I do understand that, we have 16 HFs in our environment and most of them use this built, hence my question. We were using a 3rd Party LB to manage LB activities, we are trying to get rid of that for Splunk application.

[vr2312]

@harsmarvania57 and @FrankVI

yes, it is not recommended by Splunk, but we have been running like that for the past 5 years and never came across a hiccup. These servers are highly powerful and we found them under-utilizing the resources

[FrankVl]

To better utilize server resources, you could also look into enabling multiple pipelines on a single splunk instance. Or simply replacing 16 big servers with 32 smaller servers or something. But that all depends a bit on how flexible you are in replacing servers (for virtuals that might be easier then when it is running on bare metal).

[harsmarvania57]

Best practice is, do not load-balance data transfer from S2S (Splunk to Splunk) using 3rd party LB. So you can use autoLB method as I mentioned earlier from UF to HF.

[harsmarvania57]

Hi @vr2312,

Yes you can setup auto load-balancing in outputs.conf on UF, so that UF will send data to multiple HF.

Pros:

1. It will be good to setup outputs.conf with autoLB method so that if in future any HF will go down data will still forwarded from another HF -> IDX.

As far as I know there are no cons in this auto load balancing setup.

[FrankVl · Accepted solution]

Assuming you have multiple HFs, then yes, you can configure your UFs to apply autoloadbalancing to distribute the data across those HFs. Do make sure all those HFs have the relevant TAs installed for the index time configurations (props and transforms).

You might want to take a look at the EVENT_BREAKER setting on UFs, to help them recognize event boundaries which significantly improves their autoloadbalancing behavior.

Advantage of applying load balancing already between UF and HF is that it should improve data distribution (and as a result, search performance) and it also prevents downtime of one of the HFs to block all UFs that were sending to it (through load balancing they can simply switch over to the other HFs).