topic Re: Automatic removal of duplicate log entries in Getting Data In

Automatic removal of duplicate log entries

tpride — Tue, 22 Apr 2014 00:18:16 GMT

Hi,

I currently have the following configuration:

                       --> rsyslog server (with splunk forwarder) --
                     /                                               \
Many linux Servers --                                                 --> Splunk Indexer/Search Head
                     \                                               /
                       --> rsyslog server (with splunk forwarder) --

All Linux servers have their rsyslog clients configured to forward a copy of each log entry to both of the central rsyslog servers, thus the splunk forwarders are then forwarding both copies onto the Splunk Indexder which creates a duplicate entry for each event. Given this setup is there any way of configuring Splunk to automatically remove the duplicate log entries this setup is generating (aside from disabling one of the splunk forwarders on one of the rsyslog servers)

Cheers,
Tom

Re: Automatic removal of duplicate log entries

yannK — Tue, 22 Apr 2014 01:38:41 GMT

No because the events cannot be compared to each other before being indexed. You should stop one of the sources.

At search time you can remove duplicated using "dedup" but this will not reduce your indexed volume.

Re: Automatic removal of duplicate log entries

tpride — Tue, 22 Apr 2014 06:10:25 GMT

Thanks yannK,

I pretty much expected that that would be the answer, but I needed to check because this is my first time using Splunk so I'm not up to speed on all of it's capabilities.

Cheers,
Tom