Getting Data In

Load Balancing at UF to HF

vr2312
Contributor

We have the current infrastructure :

UF -> HF -> Indexers

Can i set up Load Balancing at the outputs.conf so that data is forwarded equally to the HF ?

I would like to know the pros and cons for this.

0 Karma
1 Solution

FrankVl
Ultra Champion

Assuming you have multiple HFs, then yes, you can configure your UFs to apply autoloadbalancing to distribute the data across those HFs. Do make sure all those HFs have the relevant TAs installed for the index time configurations (props and transforms).

You might want to take a look at the EVENT_BREAKER setting on UFs, to help them recognize event boundaries which significantly improves their autoloadbalancing behavior.

Advantage of applying load balancing already between UF and HF is that it should improve data distribution (and as a result, search performance) and it also prevents downtime of one of the HFs to block all UFs that were sending to it (through load balancing they can simply switch over to the other HFs).

View solution in original post

vr2312
Contributor

Thank you @harsmarvania57 and @FrankVI for your answers.

If i have one of my Server hosting 4 instances of HF, would i still be able to achieve this ?

0 Karma

FrankVl
Ultra Champion

Theoretically you could, by having those instances use different ports or bind them each to specific (virtual) ip addresses or something like that, but running multiple instances of Splunk on a single server is not supported by Splunk, so I wouldn't recommend doing that.

What was your intention with setting up 4 HFs on a single server?

harsmarvania57
SplunkTrust
SplunkTrust

If I am understanding your comment correctly, you are running single server with 4 different splunk instances running on same server and acting as HF, in that case you can achieve this because your all HF listening/receiving data from UF on different ports but I can't see any benefit for this one because if your server will go down then all 4 HF instances will go down and UF->HF data transfer will be stopped.

Any specific reason to run 4 different splunk instances on same server because it is not a good practice.

Please correct me if I misunderstood your comment.

vr2312
Contributor

@harsmarvania57

Yes. You got me right.

I do understand that, we have 16 HFs in our environment and most of them use this built, hence my question. We were using a 3rd Party LB to manage LB activities, we are trying to get rid of that for Splunk application.

0 Karma

vr2312
Contributor

@harsmarvania57 and @FrankVI

yes, it is not recommended by Splunk, but we have been running like that for the past 5 years and never came across a hiccup. These servers are highly powerful and we found them under-utilizing the resources

0 Karma

FrankVl
Ultra Champion

To better utilize server resources, you could also look into enabling multiple pipelines on a single splunk instance. Or simply replacing 16 big servers with 32 smaller servers or something. But that all depends a bit on how flexible you are in replacing servers (for virtuals that might be easier then when it is running on bare metal).

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Best practice is, do not load-balance data transfer from S2S (Splunk to Splunk) using 3rd party LB. So you can use autoLB method as I mentioned earlier from UF to HF.

harsmarvania57
SplunkTrust
SplunkTrust

Hi @vr2312,

Yes you can setup auto load-balancing in outputs.conf on UF, so that UF will send data to multiple HF.

Pros:

  1. It will be good to setup outputs.conf with autoLB method so that if in future any HF will go down data will still forwarded from another HF -> IDX.

As far as I know there are no cons in this auto load balancing setup.

FrankVl
Ultra Champion

Assuming you have multiple HFs, then yes, you can configure your UFs to apply autoloadbalancing to distribute the data across those HFs. Do make sure all those HFs have the relevant TAs installed for the index time configurations (props and transforms).

You might want to take a look at the EVENT_BREAKER setting on UFs, to help them recognize event boundaries which significantly improves their autoloadbalancing behavior.

Advantage of applying load balancing already between UF and HF is that it should improve data distribution (and as a result, search performance) and it also prevents downtime of one of the HFs to block all UFs that were sending to it (through load balancing they can simply switch over to the other HFs).

Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...