Getting Data In

Syslog is Creating Duplicates, Prevent Splunk from ingesting duplicates

krutika_ag
Path Finder

Hi All,

 

I recently found out that my Syslog Server is creating duplicates for all log files.

I checked the packets ´pcap´ from one host and it contains unique logs but syslog has duplicates.

How do I prevent syslog from creating duplicate logs?

Is there a way to prevent Splunk from ingesting duplicate logs?


#syslog #linux #duplicates 

Labels (1)
0 Karma

JohnEGones
Communicator

@krutika_ag 

Maybe I don't entirely understand your scenario. Is there only one syslog server, or multiple ones? The syslog server, if it is properly configured does not just create duplicate entries. Check your syslog configuration both on the server and the sending nodes.

As far as ensuring that the ingestion is unique, add a CRC salt and/or ensure there is a stanza in your inputs.conf that is ignoring older files. There is a relevant discussion here:

How to avoid reindexing files after setting crcSal... - Splunk Community

inputs.conf - Splunk Documentation

0 Karma
Get Updates on the Splunk Community!

Get Operational Insights Quickly with Natural Language on the Splunk Platform

In today’s fast-paced digital world, turning data into actionable insights is essential for success. With ...

What’s New in Splunk Observability Cloud – June 2025

What’s New in Splunk Observability Cloud – June 2025 We are excited to announce the latest enhancements to ...

Almost Too Eventful Assurance: Part 2

Work While You SleepBefore you can rely on any autonomous remediation measures, you need to close the loop ...