topic Re: Recommended load balancer for indexer in Getting Data In

Recommended load balancer for indexer

sympatiko — Tue, 18 Nov 2014 14:49:59 GMT

Hi splunkers,

I just want to ask for any recommended or even tested loadbalancer upon forwarding logs to 3 indexers. My current setup is I have 3 indexers and I need my fortigate,cisco switch and some linux box logs to be forwarder on those 3 indexers in a round robin balancing.

Thanks,

Re: Recommended load balancer for indexer

lmyrefelt — Tue, 18 Nov 2014 15:14:52 GMT

You will find that it is well-documented here;

http://docs.splunk.com/Documentation/Splunk/6.2.0/Forwarding/Setuploadbalancingd

Use the built in functions in the splunk forwarder to loadbalance your data or with dns .

Re: Recommended load balancer for indexer

sympatiko — Tue, 18 Nov 2014 16:10:46 GMT

Hi Imyrefelt,

Thanks for your help. So the easiest way is to add entry to dns list? Is there is a certain setup aside from dns wherein all my logs from fortigate,cisco switches and other linux box? Because originally I have this setup wherein I used HAproxy to loadbalance the forwarded logs coming from my linux servers having a splunk forwarder to my two indexers. And I have no problem with it. Now I'm trying to involve my fortigate firewall logs and I'm experiencing this error "ERROR TcpInputProc - Received unexpected 1380997408 byte message (Invalid payload_size=1380997408 received while in parseState=1)!"

Thanks for the help!

Re: Recommended load balancer for indexer

lmyrefelt — Thu, 20 Nov 2014 09:09:52 GMT

In my opion the easist way is to use the built in function to loadbalance.

However .. in your case it seems that if you have an working load-balanced syslog solution using HaProxy and syslog-ng / what-ever ... that this might be a good solution and the only thing you need to do is install the splunk forwarder on your cluster nodes (aktive-active or passive-active?) and have them ingest the data/log-files create from syslog.

You then use the built in function in splunk forwarder to load-balance the data into your splunk indexer(-cluster)

The documentation clearly states "dont use 3dje party / hardware loadbalancer" between.

Re: Recommended load balancer for indexer

lmyrefelt — Thu, 20 Nov 2014 09:10:39 GMT

which btw is not the same as that it would not work .. just not supported

Re: Recommended load balancer for indexer

sympatiko — Thu, 20 Nov 2014 09:23:28 GMT

So this will be may overview according to your comment.

fortigate > splunk forwarder > cluster indexers

Is it?

Re: Recommended load balancer for indexer

sympatiko — Thu, 20 Nov 2014 09:45:02 GMT

Based on this http://docs.splunk.com/Documentation/Splunk/5.0/Deploy/Configureforwarderswithoutputs.confd#Load_balancing

"The forwarder will load balance between the three receivers listed. If one receiver goes down, the forwarder automatically switches to the next one available."

Somewhat confusing. Why still saying "If one receiver goes down, the forwarder automatically switches to the next one available" if it is load balancing.

Re: Recommended load balancer for indexer

lmyrefelt — Fri, 21 Nov 2014 12:59:22 GMT

fortigate > SPlunk forwarder > balance on splunk indexers .. will work. You should be able to use the addon on apps.splunk.com to get it going.

or you can use something

fortigate > Syslog-server > file > splunk-forwarder > loadbalance on indexers.