Depends on your infrastructure, but one way which we use is to set up a virtual IP on a network device that you send all log data to, which can then map to multiple physical addresses. If one of the physical addresses in the pool goes down, it'll still send to the ones remaining.
Syslog device server can send upto 4IP Address in parallel for High Availability of Forwarders but thats not the case we are looking for.
If Primary FWD goes then it must switch to Secondary FWD.
You're missing what he said. You have an intervening device presenting a static virtual IP, which (as with the F5 load-balancer suggested in the previous answer) monitors the health of the various available forwarders, and routes the traffic accordingly.
Remember, by default syslog is UDP - i.e. stateless. As remarked, something needs to be present on the indexer to indicate the health of the syslog inbound on each host.
Here my 2cents:
HeavyForwarders don't support it. As alternative solution I suggest one of below:
Use the syslog destination as a FQN A/CNAME in the DNS and update it if you need to failover - Might take some time til the DNS changes to replicate
Use a Virtual IP and a Cluster software, something like Linux heartbeat
If you have a device like F5 in your network, configure the virtual IP and fail-over rules there
Send syslog to both Splunk and perform a Dedup before index the data - waste of bandwidth/load?
Send syslog to both Splunk but one of the destinations will be off-line or blocked via firewall - you might have duplicated data if you end-up with both up at same time. Also similar bandwidth issue as above.
I'm probably going to have a F5 device in front of my heavy forwarders. The syslogs messages will be send via UDP from the devices but the F5 check rule will be checking the TCP/514 to confirm the service is up. I believe F5 will also allow me to create affinity rules....
Let me know if you have any other idea... I have similar problem with DBX App, that I still finding a better solution instead o manually migrate the configuration in case of failure.
Nice list of options 😉
Just one more to add: Linux Heartbeat can do this too
But remember, this Question is not related to Splunk. This feature must be configured outside Splunk.
For quick response.
Kindly correct me if i am wrong,
can i achieve it with check sum redundancy in inputs.conf
crcSalt = <-source>
As perimeter device(Syslog) can send syslog logs to up-to 4IP's/HFW's.
it avoids the duplicate data while Re-indexing.
With regards to #1, I changed the A record in DNS and splunk never resolves it again. It requires a reboot to updates resolution. With tcpout, you can set dnsresolutioninterval , but not with syslog:
Invalid key in stanza [syslog:mysyslog] in /opt/splunk/etc/system/local/outputs.conf, line 13: dnsresolutioninterval (value: 300)