I have batch input
[batch://C:\abc\*.zip] move_policy = sinkhole index = xyz host_segment = 2 crcSalt = <SOURCE> sourcetype = pqr disabled = false
for testing I added one zip file in monitored folder after consumed by splunk I again added same file in monitored folder and I found duplicate events. I was assumed that it will not index same file since I have included
crcSalt=<SOURCE>. What can be done avoid duplication?
Note- file monitored is zip- csv file with headers.
file names in zip files are the same or different?
both the times you had files in zip?
crcSalt=<SOURCE> guarantees that you don't index twice files with the same name, but if you have the same filename with a different path you have two files.
I am manually copying same zip file to monitor directory and number of times I am pasting files in monitored folder same number of times it is duplicating events with same source.
and zip file name and inside file name are same .