Getting Data In

Syslog is Creating Duplicates, Prevent Splunk from ingesting duplicates

krutika_ag
Path Finder

Hi All,

 

I recently found out that my Syslog Server is creating duplicates for all log files.

I checked the packets ´pcap´ from one host and it contains unique logs but syslog has duplicates.

How do I prevent syslog from creating duplicate logs?

Is there a way to prevent Splunk from ingesting duplicate logs?


#syslog #linux #duplicates 

Labels (1)
0 Karma

JohnEGones
Communicator

@krutika_ag 

Maybe I don't entirely understand your scenario. Is there only one syslog server, or multiple ones? The syslog server, if it is properly configured does not just create duplicate entries. Check your syslog configuration both on the server and the sending nodes.

As far as ensuring that the ingestion is unique, add a CRC salt and/or ensure there is a stanza in your inputs.conf that is ignoring older files. There is a relevant discussion here:

How to avoid reindexing files after setting crcSal... - Splunk Community

inputs.conf - Splunk Documentation

0 Karma
Get Updates on the Splunk Community!

Developer Spotlight with William Searle

The Splunk Guy: A Developer’s Path from Web to Cloud William is a Splunk Professional Services Consultant with ...

Major Splunk Upgrade – Prepare your Environment for Splunk 10 Now!

Attention App Developers: Test Your Apps with the Splunk 10.0 Beta and Ensure Compatibility Before the ...

Stay Connected: Your Guide to June Tech Talks, Office Hours, and Webinars!

What are Community Office Hours?Community Office Hours is an interactive 60-minute Zoom series where ...