Getting Data In

How to avoid / delete duplicate events using routers logging to central syslog

sonicZ
Contributor

Currently we are logging all our network device data from our routers to a single syslog host.
This syslog host forward to a central syslog logger which our splunk indexer monitors directly.

However we would like to log to multiple syslog hosts from the routers instead of just one but this would cause a lot of duplicate entries in our central syslogger. Anyone have a good approach to handle routers logging to multiple syslog hosts(for redundancy) but filtering duplicates before they index into the Splunk indexers?

Would rather not just pipe to dedup

results | dedup 

hopefully there is a solution to throw away dupes or an entirely new approach.

1 Solution

dwaddle
SplunkTrust
SplunkTrust

Well, Splunk itself isn't going to be able to know those events coming from different syslog servers are actually duplicates. So, there's no real way (within Splunk) to avoid the duplication.

One viable alternative is to cluster your syslog servers - use a floating IP address between the two (Red Hat's piranha / pulse comes to mind) and send all of your log data to the floating IP. Then you keep your high availability, but with only one copy of each event.

View solution in original post

dwaddle
SplunkTrust
SplunkTrust

Well, Splunk itself isn't going to be able to know those events coming from different syslog servers are actually duplicates. So, there's no real way (within Splunk) to avoid the duplication.

One viable alternative is to cluster your syslog servers - use a floating IP address between the two (Red Hat's piranha / pulse comes to mind) and send all of your log data to the floating IP. Then you keep your high availability, but with only one copy of each event.

dwaddle
SplunkTrust
SplunkTrust

I've never used haproxy/keepalived but I think that for practical matters here they'd function similarly.

0 Karma

sonicZ
Contributor

Thanks for the info Dwaddle, We were thinking of using haproxy and keep aliveD on two different syslog servers basically doing a software VIP load balanced. I'll check into piranha / pulse too.

0 Karma

jgedeon120
Contributor

You can collect your logs to as many syslog servers and have those send to a central syslog server then have the central syslog server send to Splunk. Syslog-ng is very configurable.

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Accelerate Splunk App Development with AI

  Learn how to automate the repetitive parts of your build cycle.   Watch On-Demand   Replay the Accelerate ...

Splunk Agent Observability: Measuring AI Value with Splunk Tokenomics on AWS

The AI landscape is moving faster than ever, and our latest 2026 State of Observability report provides the ...

Enterprise Security Watch Party: What’s New in 8.8 - Recap

Splunk Community HQ hosted a Splunk Enterprise Security Watch Party: What’s New In ES 8.8 last Thursday. I had ...