Load Balancer

[splunklearner]

My team has created production environment with 6 syslog servers (2 in each of 3 multi site cluster). 

My question is do two syslog servers be active active or one active and one stand by? Which will be the good practice? 

And do load balancer needs here for syslog servers? Currently some app teams are using UDP and some are TCP. basically these are network logs from network devices.

Differences bw DNS load balancer and LTM load balancer? Which is best?

Please suggest what will be the good practice to achieve this without any data loss? 

From syslog servers we have UF installed on it and forward it to our indexer.

[gcusello]

Hi @splunklearner ,

I usually configure all HFs as active and receiving syslogs (no passive HFs), in this way isn't relevent if one HF is down;  but remember that a syslog source sends logs to an HF until it receives, so, if you have a very big syslog data source it will send syslogs to only one HF at a time, because LB cannot balance traffic, but anyway the LB assures the fail over.

It's always better to have a LB in front of HF syslogs receivers to distribute load and manage fail over indipendently from the used protocol and the source.

DNS can be used as LB only if you don't have a LB because when a receiver is down, it notices late that an HF is down than a LB like F5, and in this case you loose sone logs.

If you have an LB in front of your syslog receivers you don't loss any log.

Only one hint from Best practibes: don't use Splunk as syslog receiver but rsyslog (better) or syslog-ng that writes logs in files that Splunk can read; in this way you don't need an HF but you can use an UF and your server can receive logs also when Splunk is down, in addition the load on the system is less that Splunk syslog receiving.

Ciao.

Giuseppe

[isoutamo]

Hi

1st as you are using UDP as transmit protocol you will definitely lost events. You cannot do anything for it as it due to that protocol.

You should build separate syslog cluster with VIP address and then send syslog events from those backends to splunk. Both rsyslog and syslog-ng are suitable for that. If you haven't enough experience about syslog server then probably the easiest way to achieve this is use Splunk's SC4S. You could find it from https://splunk.github.io/splunk-connect-for-syslog/main/ https://splunkbase.splunk.com/app/4740

There is also some .conf presentation about it. Probably 2020 (or 2019)?

And never use any HF or indexer as terminating TCP/UDP syslog feed with Splunk. Use always separate syslog server.

r. Ismo

[dural_yyz]

I have experience with multisite clusters so ran a single VIP which LTM to the sites, each site then LTM to the cluster of local servers.  This meant half my syslog messages crossed data centers which some would argue is not ideal but policy required site and local redundancy for the design.  Since you have UF installed you should be good from there.  This is assuming all UDP traffic which is easy to just LTM 50/50 split. DO NOT DUPLICATE messages.  Let Splunk replication factor do that for you.

The TCP part will be a bit different since you need to consider things like session length and messages per session.  A load balancer will only balance different sessions so maybe something like choosing least used link?

I don't have experience with TCP syslog so that's the best I can do for advice.