topic Re: load balancer in Deployment Architecture

load balancer

hazem — Wed, 18 Sep 2024 11:23:21 GMT

Is there any documentation in Splunk's documentation to guide a load balancer administrator on configuring the load balancer in front of intermediate forwarders to receive syslog traffic from security devices on port 514?

Re: load balancer

PickleRick — Wed, 18 Sep 2024 12:01:10 GMT

There is no such document because generally it's not recommended to LB "syslog" traffic.

You should keep your syslog receiver as simple as possible and as close to the source as possible.

Re: load balancer

richgalloway — Wed, 18 Sep 2024 12:12:26 GMT

Splunk recommendation is to NOT send syslog data directly (or via a LB) to a Splunk instance. Syslog should be sent to a dedicated syslog server (running syslog-ng or rsyslog) and then forwarded to Splunk. The syslog servers should be positioned as close to the data source as possible to avoid data loss. Use of a load balancer in front of the syslog servers is recommended for resiliency.

For more information, see

https://docs.splunk.com/Documentation/Splunk/latest/Data/HowSplunkEnterprisehandlessyslogdata#Caveats_to_using_Splunk_Enterprise_as_a_syslog_server_or_message_sender

https://www.splunk.com/en_us/blog/tips-and-tricks/high-performance-syslogging-for-splunk-using-syslog-ng-part-1.html

Re: load balancer

gcusello — Wed, 18 Sep 2024 12:14:01 GMT

Hi @hazem ,

adding a bit to @PickleRick information:

you can configure rsyslog (or syslog-ng) server on your UFs: you don't need to install it because it's already installe, you have only to configure it to understand where tp write logs.

for more infos see at https://www.rsyslog.com/guides/ or https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/6/html/deployment_guide/s1-basic_configuration_of_rsyslog

on the LB, you need only to configure the receiving port and the destination port and addresses.

Some LBs need also to configure a way to check if the destinations are alive, but this configuration depends on your LB and it's indipendent by Splunk or rsyslog receiver.

Ciao.

Giuseppe

Re: load balancer

PickleRick — Wed, 18 Sep 2024 12:19:41 GMT

Let me disagree here with you on one thing. Adding a load balancer in front of syslog receivers does not usually solve any problems (especially because LBs typically "don't speak" syslog; and even more so since "syslog" can mean many different things - from RFC5424-compliant message to "just throw anyting at UDP/514") and introduces additional layer of complexity and a potential SPOF.

Re: load balancer

hazem — Wed, 18 Sep 2024 12:35:21 GMT

Dear @gcusello

I have already configured rsyslog on both intermediate forwarders and need to set up the load balancer to receive traffic from syslog devices and forward it to a single backend intermediate forwarder. If the load balancer administrator asks, what is the best practice for configuring the load balancer to forward traffic to our intermediate forwarder?

Re: load balancer

gcusello — Wed, 18 Sep 2024 13:21:53 GMT

Hi @hazem ,

as I said,

On the LB you have only to configure the rule to associate the receiving port with the ip addresses and port of the receivers.

In addition, depending on the LB, you should configure how the LB checks if the receivers are alive, but this isn't a Splunk configuration and it depends on the LB (and I cannot help you.

In other word: you must define a VIP and a port to use to send logs from the syslog sources, and then associate these VIP and port to the destination IP addresses and port (of the UFs.

There isn't a best practice, only that the LB must check if the destinations are alive.

There's only one not clear thing: why are you speaking of a single intermediate Forwarder?

To have HA, you need at least two UFs, otherwise the LB is completely useless.

Ciao.

Giuseppe

Re: load balancer

hazem — Thu, 19 Sep 2024 08:05:29 GMT

Hello @gcusello

regarding your question:

There's only one not clear thing: why are you speaking of a single intermediate Forwarder?

No, I have 2 forwarders, but as you know, since UDP is a stream, one forwarder will handle all traffic.

Re: load balancer

gcusello — Thu, 19 Sep 2024 08:09:39 GMT

Hi @hazem ,

ok, not it's clear.

Anyway,

let us know if we can help you more, or, please, accept one answer for the other people of Community.

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the contributors 😉

Re: load balancer

PickleRick — Thu, 19 Sep 2024 08:23:40 GMT

Actually, UDP is _not_ a stream. UDP is a connectionless protocol and every datagram is independent from all other ones.

Re: load balancer

gcusello — Thu, 19 Sep 2024 10:03:56 GMT

Hi @hazem ,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the contributors 😉

Re: load balancer

hazem — Sun, 29 Sep 2024 07:45:58 GMT

Hello @gcusello

please advise the load balancing method connectivity from source IP to go to the forwarder node .. the persistency or we can keep it round robin?

we have 2 forwarders

Re: load balancer

isoutamo — Sun, 29 Sep 2024 08:37:11 GMT

As most other configuration items this also has answer it depends on your environment. There are some known issues with some vendors and as @PickleRick said, pure syslog traffic with UDP is not good candidate for LB. But e.g. if you are using rsyslog with RELP protocol then it's totally different case. You could use e.g. F5 in front of rsyslog backends and this works well after you have select e.g. FastL4 profile for LB. Without it you will lost some events.

r. Ismo

Re: load balancer

isoutamo — Sun, 29 Sep 2024 08:39:29 GMT

That's true when you are using UDP and pure syslog. But fortunately usually you can select other transform protocol that UDP for syslog or even use rsyslog with RELP. In those cases I strongly recommend to use LB with correct configurations.

Re: load balancer

isoutamo — Sun, 29 Sep 2024 08:41:59 GMT

As I said, this depends on LB. In some vendors you will lose event if LB check if backend is up or down, and if it is down then e.g. F5 just drop the packet (or at least it done it couple of years ago when I last use it). You must change the profile to get it to work correctly.

Re: load balancer

hazem — Sun, 29 Sep 2024 09:09:19 GMT

hello @isoutamo

please advise the load balancing method connectivity from source IP to go to the forwarder node .. the persistency or we can keep it round robin?

we have 2 forwarders

Re: load balancer

isoutamo — Sun, 29 Sep 2024 09:24:48 GMT

As I said, this depends on e.g. are you using plain syslog, udp vs tcp vs tsl etc.
If you are using udp there is no matter what you are doing, you will lose some events anyway.

Re: load balancer

gcusello — Sun, 29 Sep 2024 14:55:11 GMT

Hi @hazem ,

I'm not an exper in Load Balancers, and, as @isoutamo said, it depends on the Load Balancer: ask this question to a specialist of your LB.

Ciao.

Giuseppe.

P.S.: Karma Points are appreciated by all the contributors 😉