Getting Data In

Duplicate event with rotate log

dungnq
Loves-to-Learn

Hi team,

I encountered a problem when retrieving data from rotate log files: duplicate event.
For example: the event in file test.log.1 has been retrieved, when rotating to test.log.2 splunk retrieves it again.
How to configure splunk to only retrieve the latest events and not events that have been rotated to another file?
=====
Log4j App information:
log4j.appender.file.File=test.log
log4j.appender.file.MaxFileSize=10000KB
log4j.appender.file.MaxBackupIndex=99
=====
Splunk inputs.conf information
[monitor:///opt/IBM/WebSphere/AppServer/profiles/APP/test.log*]

Labels (2)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @dungnq,

are you using crcSalt = <SOURCE> in your inputs.conf?

Ciao.

Giuseppe

0 Karma

dungnq
Loves-to-Learn

Hi @gcusello,
Thank you for your response. Currently I have configured the crcSalt parameter on the inputs.conf file
----

[monitor:///opt/IBM/WebSphere/AppServer/profiles/APP/test.log*]
index = app
sourcetype = tws:aws:testdev:app
disabled = false
crcSalt = <SOURCE>
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @dungnq,

is it mandatory for your ingestion?

this is the reason for the double ingestion: logs arrive from different files.

Without crcSalt = <SOURCE>, Splunk doesn't index twice a log.

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

Earn a $35 Gift Card for Answering our Splunk Admins & App Developer Survey

Survey for Splunk Admins and App Developers is open now! | Earn a $35 gift card!      Hello there,  Splunk ...

Continuing Innovation & New Integrations Unlock Full Stack Observability For Your ...

You’ve probably heard the latest about AppDynamics joining the Splunk Observability portfolio, deepening our ...

Monitoring Amazon Elastic Kubernetes Service (EKS)

As we’ve seen, integrating Kubernetes environments with Splunk Observability Cloud is a quick and easy way to ...