Automatic removal of duplicate log entries

[tpride]

Hi,

I currently have the following configuration:

                       --> rsyslog server (with splunk forwarder) --
                     /                                               \
Many linux Servers --                                                 --> Splunk Indexer/Search Head
                     \                                               /
                       --> rsyslog server (with splunk forwarder) --

All Linux servers have their rsyslog clients configured to forward a copy of each log entry to both of the central rsyslog servers, thus the splunk forwarders are then forwarding both copies onto the Splunk Indexder which creates a duplicate entry for each event. Given this setup is there any way of configuring Splunk to automatically remove the duplicate log entries this setup is generating (aside from disabling one of the splunk forwarders on one of the rsyslog servers)

Cheers,
Tom

[yannK · Accepted solution]

No because the events cannot be compared to each other before being indexed. You should stop one of the sources.

At search time you can remove duplicated using "dedup" but this will not reduce your indexed volume.

[tpride]

Thanks yannK,

I pretty much expected that that would be the answer, but I needed to check because this is my first time using Splunk so I'm not up to speed on all of it's capabilities.

Cheers,
Tom