How to avoid / delete duplicate events using routers logging to central syslog

[sonicZ]

Currently we are logging all our network device data from our routers to a single syslog host.
This syslog host forward to a central syslog logger which our splunk indexer monitors directly.

However we would like to log to multiple syslog hosts from the routers instead of just one but this would cause a lot of duplicate entries in our central syslogger. Anyone have a good approach to handle routers logging to multiple syslog hosts(for redundancy) but filtering duplicates before they index into the Splunk indexers?

Would rather not just pipe to dedup

results | dedup 

hopefully there is a solution to throw away dupes or an entirely new approach.

[dwaddle · Accepted solution]

Well, Splunk itself isn't going to be able to know those events coming from different syslog servers are actually duplicates. So, there's no real way (within Splunk) to avoid the duplication.

One viable alternative is to cluster your syslog servers - use a floating IP address between the two (Red Hat's piranha / pulse comes to mind) and send all of your log data to the floating IP. Then you keep your high availability, but with only one copy of each event.

[dwaddle]

I've never used haproxy/keepalived but I think that for practical matters here they'd function similarly.

[sonicZ]

Thanks for the info Dwaddle, We were thinking of using haproxy and keep aliveD on two different syslog servers basically doing a software VIP load balanced. I'll check into piranha / pulse too.

[jgedeon120]

You can collect your logs to as many syslog servers and have those send to a central syslog server then have the central syslog server send to Splunk. Syslog-ng is very configurable.