Deployment Architecture

Avoid indexing same file multiple times batch input

ips_mandar
Builder

I have batch input

[batch://C:\abc\*.zip]
move_policy = sinkhole
index = xyz
host_segment = 2
crcSalt = <SOURCE>
sourcetype = pqr
disabled = false

for testing I added one zip file in monitored folder after consumed by splunk I again added same file in monitored folder and I found duplicate events. I was assumed that it will not index same file since I have included crcSalt=<SOURCE>. What can be done avoid duplication?

Note- file monitored is zip- csv file with headers.

Tags (1)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi ips_mandar
file names in zip files are the same or different?
both the times you had files in zip?
crcSalt=<SOURCE> guarantees that you don't index twice files with the same name, but if you have the same filename with a different path you have two files.

Bye.
Giuseppe

0 Karma

ips_mandar
Builder

Hi @gcusello,
I am manually copying same zip file to monitor directory and number of times I am pasting files in monitored folder same number of times it is duplicating events with same source.
and zip file name and inside file name are same .

0 Karma

ips_mandar
Builder

Not sure if it works for batch input since it works for monitor input.

0 Karma

ips_mandar
Builder

Any idea anyone to avoid indexing same file multiple time in batch input?

0 Karma
Get Updates on the Splunk Community!

Dashboards: Hiding charts while search is being executed and other uses for tokens

There are a couple of features of SimpleXML / Classic dashboards that can be used to enhance the user ...

Splunk Observability Cloud's AI Assistant in Action Series: Explaining Metrics and ...

This is the fourth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how ...

Brains, Bytes, and Boston: Learn from the Best at .conf25

When you think of Boston, you might picture colonial charm, world-class universities, or even the crack of a ...