Forwarder High Availability solution

[xshen_anji]

We have a Forwarder deloyed to accept syslog and SNMP traffic and forward them to multiple indexers. If the forwarder is just one a machine, the unexpected shutdown of the machine will break the data transfer. We would like to have a HA solution on the forwarder to make sure the service uninterruptable. The forwarder now is running CentOS 7. We searched for several HA open source solutions, like pcs, Linux-HA, but not sure which one is reliable, could anybody give some suggestions?

[nickhills]

The challenge is that by default both syslog and SNMP do not inherently lend themselves to HA solutions.

Both 'tend' to use UDP as the transport and this makes reliable (guaranteed) transmission and receipt difficult to evidence.
Some vendors allow you to select TCP as a transport protocol, and configure multiple destinations. For these products the sending device may retry to send event data if the original transmission is not acknowledged, but delivery is still not guaranteed.

The solutions available to you are therefore:

a.) Build an HA(ish) syslog-ng/snmpd host and couple this with network load balancers and some tight monitoring.
Then use a UF to collect the snmpd/syslog events centrally and send for indexing.

b.) Where your endpoints are servers/anything other than network devices/legacy endpoints that only support syslog, migrate away from using syslog/snmp to a Splunk UF on each endpoint with local log monitoring and metric collection with something like collectd

[lakshman239]

One option would be to have another forwarder ( with syslog-ng or rsyslog for receiving data) and front-ended with a load balancer. So, the data from your devices will hit the load balancer and then send them via round robin to both forwarders. So, in the event of upgrade/availability, there is atleast one forwarder avaiable.