Load Balancing Splunk using F5 Load balancer

[sankaraniyan1]

I have a client requirement to use F5 Big IP LB for load-balancing the splunk data collection. Can anyone help me with the best /recommended method to do health check for Splunk load balancing at Indexer level. Is it using http/https or is it better to use TCP based ?
Also what type of policy will be better to use ? Round robin with String based or something different ?
Please help me with your kind suggestions based on the experience .

Sank

[richgalloway]

Do not use a load balancer of any kind in front of Splunk indexers. Splunk uses a proprietary protocol that load balancers don't support.
You should send data to the indexers using a Splunk forwarder, which can be configured to distribute load evenly among indexers.

[sankaraniyan1]

Thanks for the response Richgalloway. Is this recommended by Splunk or is it from your implementation experience. May i know any challenges you faced ?

Sank

[richgalloway]

@gcusello gave you the link to the documentation. I have experience with customers wanting to do this so I tell them they will not like the results and they don't to it.

[gcusello]

Hi @sankaraniyan1,
if you're speaking of logs from Universal Forwarders, you don't need a Load balancer because Splunk already automatically manages Load Banacing and Fail Over.

if instead you are speaking of syslogs or HEC, it's a good idea and you can create a VIP and use it to send syslogs and/or HEC and use F5 to distribute them to Heavy Forwarders o Indexers.

About Health Check, you can generate an hearth beat from Splunk (using an alert) if your F5 hasn't any other method to check the status of destinations.

About protocol, use TCP if possible or http/https for HEC or UDP for syslogs.

At least, you can use F5 if you have a Search Head Cluster to distribute users between SHs.

You can find more infos at https://docs.splunk.com/Documentation/Splunk/8.0.3/Forwarding/Setuploadbalancingd .

Ciao.
Giuseppe

[sankaraniyan1]

Hi gcusello,
We are using our forwarders mainly for metric data collection like cpu, memory and other applications using Add-on Apps. Ideally client team expects to use an external LB instead of the default LB functionality provided by Splunk. I couldn't find any information in splunk documentation. I would like to understand if this is configurable and the best practice or if not what are the issues .

[gcusello]

Hi @sankaraniyan1,
as me and @richgalloway said, it isn't a good idea because Splunk uses an internal and proprietary protocol to manage Load Balancing, so if you use an external tool, you need to manage all the states of the components, when you already have already integrated and ready all you need!

As I said, your should use a Load Balancer if you have syslogs or HEC and for user front-end, not for forwarding data from Forwarders and Indexers.

You can find more infos at https://docs.splunk.com/Documentation/Splunk/8.0.4/Forwarding/Setuploadbalancingd : in this page is clearly written: "You should not use an external load balancer to implement load balancing between forwarders and receivers. This practice does not generate the results you would expect. Use the load balancing capability that comes with the forwarder."

Ciao.
Giuseppe

[sankaraniyan1]

Thanks Guiseppe. This helps. In case i need to enable load balancing at user front-end, what will be the health monitoring string we could configure for the load balancer ?
Any recommendations will be appreciated

Sank

[gcusello · Accepted solution]

Hi @sankaraniyan1,
for user front-end you don't need any health monitoring, only a virtual address to use for accessing.

Maybe these urls can help you:
https://www.splunk.com/en_us/blog/tips-and-tricks/configuring-nginx-load-balancer-for-the-http-event...
http://nginx.org/en/docs/http/load_balancing.html#nginx_load_balancing_configuration

Ciao.
Giuseppe