https://community.splunk.com/t5/Getting-Data-In/How-to-find-a-solution-to-not-have-duplicated-logs/m-p/598687#M104420 <P>No. Splunk on its own does not do any form of deduplication. It's up to you to provide the input data in the form you need.</P><P>I must say however that I don't quite understand what do you mean by "2 forwarders with the same logs". Some network share and two separate clients mounting it? Why don't you then ingest the logs simply from the source machine?</P><P>&nbsp;</P> Fri, 20 May 2022 16:00:43 GMT PickleRick 2022-05-20T16:00:43Z https://community.splunk.com/t5/Getting-Data-In/How-to-find-a-solution-to-not-have-duplicated-logs/m-p/598684#M104419 <P>Hello there,</P> <P>The deal is that I have 2 forwarders that have exactly the same logs (I'm using 2 forwarders not to have a SPOF) and I want to find a solution to not have duplicated logs. I thought of using a load balancer but I just want to know first if there is some config on Splunk that allows to do that please.</P> <P>Best regards,</P> <P>&nbsp;Abir</P> Fri, 20 May 2022 16:07:20 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-find-a-solution-to-not-have-duplicated-logs/m-p/598684#M104419 aatik5u 2022-05-20T16:07:20Z https://community.splunk.com/t5/Getting-Data-In/How-to-find-a-solution-to-not-have-duplicated-logs/m-p/598687#M104420 <P>No. Splunk on its own does not do any form of deduplication. It's up to you to provide the input data in the form you need.</P><P>I must say however that I don't quite understand what do you mean by "2 forwarders with the same logs". Some network share and two separate clients mounting it? Why don't you then ingest the logs simply from the source machine?</P><P>&nbsp;</P> Fri, 20 May 2022 16:00:43 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-find-a-solution-to-not-have-duplicated-logs/m-p/598687#M104420 PickleRick 2022-05-20T16:00:43Z https://community.splunk.com/t5/Getting-Data-In/How-to-find-a-solution-to-not-have-duplicated-logs/m-p/598753#M104424 <P>Hi&nbsp;<a href="https://community.splunk.com/t5/user/viewprofilepage/user-id/244402">@aatik5u</a>,</P><P>as&nbsp;<a href="https://community.splunk.com/t5/user/viewprofilepage/user-id/231884">@PickleRick</a>&nbsp;said, there isn't any configuration for avoid duplications.</P><P>If you're speaking of network or HEC logs and you cannot use a Load Balancer, you could configure your DNS to distribute logs to both the Forwarders and manage fails.</P><P>Here you can find how to do it&nbsp;<A href="https://docs.microsoft.com/en-us/windows-server/networking/dns/deploy/app-lb" target="_blank">https://docs.microsoft.com/en-us/windows-server/networking/dns/deploy/app-lb</A></P><P>Ciao.</P><P>Giuseppe</P> Sat, 21 May 2022 05:20:00 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-find-a-solution-to-not-have-duplicated-logs/m-p/598753#M104424 gcusello 2022-05-21T05:20:00Z https://community.splunk.com/t5/Getting-Data-In/How-to-find-a-solution-to-not-have-duplicated-logs/m-p/598829#M104435 <P>Thank you very much for your answer <span class="lia-unicode-emoji" title=":slightly_smiling_face:">🙂</span></P><P>to answer your question, I have several linux machines that forward their logs to 2 different universal forwarders using syslog, that's why I have the same logs twice. this choice is kind of questionable I agree :/, but there is no specific reason for choosing to forward logs to one universal forwarder than using a universal forwarder on every linux machine</P> Mon, 23 May 2022 06:22:06 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-find-a-solution-to-not-have-duplicated-logs/m-p/598829#M104435 aatik5u 2022-05-23T06:22:06Z https://community.splunk.com/t5/Getting-Data-In/How-to-find-a-solution-to-not-have-duplicated-logs/m-p/598830#M104436 <P>Thank you very much <span class="lia-unicode-emoji" title=":slightly_smiling_face:">🙂</span></P> Mon, 23 May 2022 06:22:57 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-find-a-solution-to-not-have-duplicated-logs/m-p/598830#M104436 aatik5u 2022-05-23T06:22:57Z