https://community.splunk.com/t5/Getting-Data-In/How-to-avoid-delete-duplicate-events-using-routers-logging-to/m-p/80035#M16466 <P>I've never used haproxy/keepalived but I think that for practical matters here they'd function similarly.</P> Tue, 28 Feb 2012 18:48:26 GMT dwaddle 2012-02-28T18:48:26Z https://community.splunk.com/t5/Getting-Data-In/How-to-avoid-delete-duplicate-events-using-routers-logging-to/m-p/80031#M16462 <P>Currently we are logging all our network device data from our routers to a single syslog host.<BR /> This syslog host forward to a central syslog logger which our splunk indexer monitors directly.</P> <P>However we would like to log to multiple syslog hosts from the routers instead of just one but this would cause a lot of duplicate entries in our central syslogger. Anyone have a good approach to handle routers logging to multiple syslog hosts(for redundancy) but filtering duplicates before they index into the Splunk indexers? </P> <P>Would rather not just pipe to dedup</P> <PRE><CODE>results | dedup </CODE></PRE> <P>hopefully there is a solution to throw away dupes or an entirely new approach.</P> Mon, 27 Feb 2012 23:39:00 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-avoid-delete-duplicate-events-using-routers-logging-to/m-p/80031#M16462 sonicZ 2012-02-27T23:39:00Z https://community.splunk.com/t5/Getting-Data-In/How-to-avoid-delete-duplicate-events-using-routers-logging-to/m-p/80032#M16463 <P>You can collect your logs to as many syslog servers and have those send to a central syslog server then have the central syslog server send to Splunk. Syslog-ng is very configurable.</P> Tue, 28 Feb 2012 00:41:59 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-avoid-delete-duplicate-events-using-routers-logging-to/m-p/80032#M16463 jgedeon120 2012-02-28T00:41:59Z https://community.splunk.com/t5/Getting-Data-In/How-to-avoid-delete-duplicate-events-using-routers-logging-to/m-p/80033#M16464 <P>Well, Splunk itself isn't going to be able to know those events coming from different syslog servers are actually duplicates. So, there's no real way (within Splunk) to avoid the duplication.</P> <P>One viable alternative is to cluster your syslog servers - use a floating IP address between the two (Red Hat's piranha / pulse comes to mind) and send all of your log data to the floating IP. Then you keep your high availability, but with only one copy of each event.</P> Tue, 28 Feb 2012 14:36:03 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-avoid-delete-duplicate-events-using-routers-logging-to/m-p/80033#M16464 dwaddle 2012-02-28T14:36:03Z https://community.splunk.com/t5/Getting-Data-In/How-to-avoid-delete-duplicate-events-using-routers-logging-to/m-p/80034#M16465 <P>Thanks for the info Dwaddle, We were thinking of using haproxy and keep aliveD on two different syslog servers basically doing a software VIP load balanced. I'll check into piranha / pulse too.</P> Tue, 28 Feb 2012 17:04:54 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-avoid-delete-duplicate-events-using-routers-logging-to/m-p/80034#M16465 sonicZ 2012-02-28T17:04:54Z https://community.splunk.com/t5/Getting-Data-In/How-to-avoid-delete-duplicate-events-using-routers-logging-to/m-p/80035#M16466 <P>I've never used haproxy/keepalived but I think that for practical matters here they'd function similarly.</P> Tue, 28 Feb 2012 18:48:26 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-avoid-delete-duplicate-events-using-routers-logging-to/m-p/80035#M16466 dwaddle 2012-02-28T18:48:26Z