Hardware load balancer between indexer and forwarder instead of auto-lb config?

[mfrost8]

Hello.

I'm working to create some forwarders in our DMZ to relay data from the DMZ to our internal indexers (currently 2 indexers) and am considering a couple of different configurations. I refer to those as Splunk DMZ gateways below.

Our regular internal forwarders have their outputs.conf configured to use auto-load balancing when sending data to the indexers. As I understand it, this configuration flips to the next indexer in outputs.conf every 30 seconds, assuming they're available. This works well for availability, but my understanding this is also nice in terms of spreading the data out more or less evenly on the indexers.

My original thought was to use this same configuration for the Splunk DMZ "gateway" servers I'm working on. This should work well for the same reasons I listed above with internal forwarders. My concern is that we will likely be adding more indexers and shuffling locations around on the internal side which means a number of firewall rule changes to accomodate those.

So I was thinking that I could create an F5 (LTM) load balancer pool and just have the DMZ gateways send to that. The pool would then be configured to have whatever indexers in it that I wanted the Splunk DMZ gateway servers to send to. In other words, I need only one firewall rule for that and it's pretty static. Changing member nodes on the load balancer pool is pretty simple. The Splunk DMZ gateway servers would not use auto-load balancing and instead send to the load balancer VIP.

While I'm pretty confident that this load balancer between the Splunk DMZ gateway and the indexers will work, I'm a little concerned that the events may not be spread quite as evenly as they might be with the auto-lb config. I think the default algorithm we use for the pool members is "least connections". We could fiddle with things like "round robin", but I'm not convinced that that would make things better.

I'm also curious what the behavior is without auto load balancing in outputs.conf. Does Splunk disconnection periodically (like it disconnects every 30 seconds with auto-lb) or does it just stay connected? If it stays connected then that might be a problem as it would not switch between load balancer pool nodes unless one went down.

Does anyone have any thoughts or experience with this load balancer config? Or are my concerns nothing to worry about?

Thanks

[Takajian]

I think you can also use intermediate splunk forwarder instead of F5 load balancer. The intermediate splunk forwader will solve your concern. Do you have reason to user F5, not intermediate forwarder?

[mfrost8]

It looks like Splunk will only disconnect/reconnect with an autoLB config. I had hoped there was an easy way to fool it into acting like that, but I can't find it. Maybe if I defined two VIPs and told output.conf to autoLB between them, but that gets ridiculous. That lack of ability to disconnect and reconnect might be a deal killer. Pity as this would buy a lot of flexibility as it's easier for us to change LB nodes (members) than it is to change firewall rules.

[Takajian]

Now I understand it clearly.I have never done such a deployment. But I do not think splunk disconnect connection periodically according to forwarded events. If you setup multiple VIP for each intermediate forwarder as one by one, it would not be problem. But it will not much your requirment. Sorry it will not help.

[mfrost8]

Thanks. I probably didn't say that very well. This thing I'm calling a DMZ gateway is an intermediate forwarder. It would sit in the DMZ and get events from hosts in the DMZ and forward those to internal indexers. I'm talking about sending traffic from this intermediate forwarder to a LB pool rather than directly to indexers. Essentially it can buy me more flexibility to send that traffic to a single, simple VIP on the LB rather than sending to one or more indexers that may change as we build out our infrastructure.