Getting Data In

Syslog is Creating Duplicates, Prevent Splunk from ingesting duplicates

krutika_ag
Path Finder

Hi All,

 

I recently found out that my Syslog Server is creating duplicates for all log files.

I checked the packets ´pcap´ from one host and it contains unique logs but syslog has duplicates.

How do I prevent syslog from creating duplicate logs?

Is there a way to prevent Splunk from ingesting duplicate logs?


#syslog #linux #duplicates 

Labels (1)
0 Karma

JohnEGones
Communicator

@krutika_ag 

Maybe I don't entirely understand your scenario. Is there only one syslog server, or multiple ones? The syslog server, if it is properly configured does not just create duplicate entries. Check your syslog configuration both on the server and the sending nodes.

As far as ensuring that the ingestion is unique, add a CRC salt and/or ensure there is a stanza in your inputs.conf that is ignoring older files. There is a relevant discussion here:

How to avoid reindexing files after setting crcSal... - Splunk Community

inputs.conf - Splunk Documentation

0 Karma
Get Updates on the Splunk Community!

CX Day is Coming!

Customer Experience (CX) Day is on October 7th!! We're so excited to bring back another day full of wonderful ...

Strengthen Your Future: A Look Back at Splunk 10 Innovations and .conf25 Highlights!

The Big One: Splunk 10 is Here!  The moment many of you have been waiting for has arrived! We are thrilled to ...

Now Offering the AI Assistant Usage Dashboard in Cloud Monitoring Console

Today, we’re excited to announce the release of a brand new AI assistant usage dashboard in Cloud Monitoring ...