load balancer

[hazem]

Is there any documentation in Splunk's documentation to guide a load balancer administrator on configuring the load balancer in front of intermediate forwarders to receive syslog traffic from security devices on port 514?

[hazem]

Dear @gcusello 

I have already configured rsyslog on both intermediate forwarders and need to set up the load balancer to receive traffic from syslog devices and forward it to a single backend intermediate forwarder. If the load balancer administrator asks, what is the best practice for configuring the load balancer to forward traffic to our intermediate forwarder?

[gcusello · Accepted solution]

Hi @hazem ,

as I said,

On the LB you have only to configure the rule to associate the receiving port with the ip addresses and port of the receivers.

In addition, depending on the LB, you should configure how the LB checks if the receivers are alive, but this isn't a Splunk configuration and it depends on the LB (and I cannot help you.

In other word: you must define a VIP and a port to use to send logs from the syslog sources, and then associate these VIP and port to the destination IP addresses and port (of the UFs.

There isn't a best practice, only that the LB must check if the destinations are alive.

There's only one not clear thing: why are you speaking of a single intermediate Forwarder?

To have HA, you need at least two UFs, otherwise the LB is completely useless.

Ciao.

Giuseppe

[isoutamo]

As I said, this depends on LB. In some vendors you will lose event if LB check if backend is up or down, and if it is down then e.g. F5 just drop the packet (or at least it done it couple of years ago when I last use it). You must change the profile to get it to work correctly.

[hazem]

Hello @gcusello 

regarding your question:

There's only one not clear thing: why are you speaking of a single intermediate Forwarder?

No, I have 2 forwarders, but as you know, since UDP is a stream, one forwarder will handle all traffic.

[gcusello]

Hi @hazem ,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the contributors 😉

[hazem]

Hello @gcusello 

please advise the load balancing method connectivity from source IP to go to the forwarder node .. the persistency or we can keep it round robin?

we have 2 forwarders

[gcusello]

Hi @hazem ,

I'm not an exper in Load Balancers, and, as @isoutamo said, it depends on the Load Balancer: ask this question to a specialist of your LB.

Ciao.

Giuseppe.

P.S.: Karma Points are appreciated by all the contributors 😉

[PickleRick]

Actually, UDP is _not_ a stream. UDP is a connectionless protocol and every datagram is independent from all other ones.

[gcusello]

Hi @hazem ,

ok, not it's clear.

Anyway, 

let us know if we can help you more, or, please, accept one answer for the other people of Community.

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the contributors 😉

[gcusello]

Hi @hazem ,

adding a bit to @PickleRick information:

you can configure rsyslog (or syslog-ng) server on your UFs: you don't need to install it because it's already installe, you have only to configure it to understand where tp write logs.

for more infos see at https://www.rsyslog.com/guides/ or https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/6/html/deployment_guide/s1-basic_c... 

on the LB, you need only to configure the receiving port and the destination port and addresses.

Some LBs need also to configure a way to check if the destinations are alive, but this configuration depends on your LB and it's indipendent by Splunk or rsyslog receiver.

Ciao.

Giuseppe

[richgalloway]

Splunk recommendation is to NOT send syslog data directly (or via a LB) to a Splunk instance.  Syslog should be sent to a dedicated syslog server (running syslog-ng or rsyslog) and then forwarded to Splunk.  The syslog servers should be positioned as close to the data source as possible to avoid data loss.  Use of a load balancer in front of the syslog servers is recommended for resiliency.

For more information, see

https://docs.splunk.com/Documentation/Splunk/latest/Data/HowSplunkEnterprisehandlessyslogdata#Caveat...

https://www.splunk.com/en_us/blog/tips-and-tricks/high-performance-syslogging-for-splunk-using-syslo...

[PickleRick]

Let me disagree here with you on one thing. Adding a load balancer in front of syslog receivers does not usually solve any problems (especially because LBs typically "don't speak" syslog; and even more so since "syslog" can mean many different things - from RFC5424-compliant message to "just throw anyting at UDP/514") and introduces additional layer of complexity and a potential SPOF.

[isoutamo]

That's true when you are using UDP and pure syslog. But fortunately usually you can select other transform protocol that UDP for syslog or even use rsyslog with RELP. In those cases I strongly recommend to use LB with correct configurations.

[hazem]

hello @isoutamo 

please advise the load balancing method connectivity from source IP to go to the forwarder node .. the persistency or we can keep it round robin?

we have 2 forwarders

[isoutamo]

As I said, this depends on e.g. are you using plain syslog, udp vs tcp vs tsl etc.
If you are using udp there is no matter what you are doing, you will lose some events anyway.

[PickleRick]

There is no such document because generally it's not recommended to LB "syslog" traffic.

You should keep your syslog receiver as simple as possible and as close to the source as possible.

[isoutamo]

As most other configuration items this also has answer it depends on your environment. There are some known issues with some vendors and as @PickleRick said, pure syslog traffic with UDP is not good candidate for LB. But e.g. if you are using rsyslog with RELP protocol then it's totally different case. You could use e.g. F5 in front of rsyslog backends and this works well after you have select e.g. FastL4 profile for LB. Without it you will lost some events.

r. Ismo