Getting Data In

If syslogs are sent to multiple forwarders, would the syslogs be duplicated in an indexer cluster?

New Member

I have multiple forwarders and an indexer cluster. If the syslogs source devices were to send syslogs to both forwarders, would the syslogs be duplicated in an index defined in the cluster or is there a way to stop that from happening?

Normally I would have suggested using a load balancer or anycast but that solution is not available for us yet. We want to ensure no syslogs are lost even if one forwarder goes down but we don't want duplicates either.

0 Karma


I spent some time researching the exact issue. In the end the team decided against any complexity at this point, so we are going with two syslog servers behind a load balancer.

When digging around, I did find a few resources that may be of interest.

One is an article from 2014, The State of Highly-Available Syslog, which also mentioned the second solution: a blog entry and an open source tool from 2013, Overengineering Syslog: Redundancy, High Availability, Deduplication, and Splunk.

I have not used duplog, so can't comment on that.

0 Karma


Back in the day, our syslog servers were set up similarly. See the answer i posted here about how we configured them. Not sure it's a good practice, but if you have nothing else?

0 Karma

Revered Legend

If you're sending same data to both forwarders, the data would get duplicated. Splunk doesn't have any ways to know if the incoming data is already indexes and avoid duplicate data indexing. This has to be sorted out before it's sent to Indexers (or forwarder for that matter). Best option was to setup a load balancer if it was available.

0 Karma
Get Updates on the Splunk Community!

Devesh Logendran, Splunk, and the Singapore Cyber Conquest

At this year’s Splunk University, I had the privilege of chatting with Devesh Logendran, one of the winners in ...

There's No Place Like Chrome and the Splunk Platform

WATCH NOW!Malware. Risky Extensions. Data Exfiltration. End-users are increasingly reliant on browsers to ...

Customer Experience | Join the Customer Advisory Board!

Are you ready to take your Splunk journey to the next level? 🚀 We invite you to join our elite squad ...