Syslog is Creating Duplicates, Prevent Splunk from ingesting duplicates

[krutika_ag]

Hi All,

I recently found out that my Syslog Server is creating duplicates for all log files.

I checked the packets ´pcap´ from one host and it contains unique logs but syslog has duplicates.

How do I prevent syslog from creating duplicate logs?

Is there a way to prevent Splunk from ingesting duplicate logs?

#syslog #linux #duplicates 

[JohnEGones]

@krutika_ag 

Maybe I don't entirely understand your scenario. Is there only one syslog server, or multiple ones? The syslog server, if it is properly configured does not just create duplicate entries. Check your syslog configuration both on the server and the sending nodes.

As far as ensuring that the ingestion is unique, add a CRC salt and/or ensure there is a stanza in your inputs.conf that is ignoring older files. There is a relevant discussion here:

How to avoid reindexing files after setting crcSal... - Splunk Community

inputs.conf - Splunk Documentation