topic Re: Duplicate event with rotate log in Getting Data In

Duplicate event with rotate log

dungnq — Fri, 17 May 2024 08:43:31 GMT

Hi team,

I encountered a problem when retrieving data from rotate log files: duplicate event.
For example: the event in file test.log.1 has been retrieved, when rotating to test.log.2 splunk retrieves it again.
How to configure splunk to only retrieve the latest events and not events that have been rotated to another file?
=====
Log4j App information:
log4j.appender.file.File=test.log
log4j.appender.file.MaxFileSize=10000KB
log4j.appender.file.MaxBackupIndex=99
=====
Splunk inputs.conf information
[monitor:///opt/IBM/WebSphere/AppServer/profiles/APP/test.log*]

Re: Duplicate event with rotate log

gcusello — Fri, 17 May 2024 08:59:26 GMT

Hi @dungnq,

are you using crcSalt = <SOURCE> in your inputs.conf?

Ciao.

Giuseppe

Re: Duplicate event with rotate log

dungnq — Fri, 17 May 2024 09:34:49 GMT

Hi @gcusello,
Thank you for your response. Currently I have configured the crcSalt parameter on the inputs.conf file
----

[monitor:///opt/IBM/WebSphere/AppServer/profiles/APP/test.log*] index = app sourcetype = tws:aws:testdev:app disabled = false crcSalt = <SOURCE>

Re: Duplicate event with rotate log

gcusello — Fri, 17 May 2024 09:35:32 GMT

Hi @dungnq,

is it mandatory for your ingestion?

this is the reason for the double ingestion: logs arrive from different files.

Without crcSalt = <SOURCE>, Splunk doesn't index twice a log.

Ciao.

Giuseppe