topic Does batch input ignore files already read? in Getting Data In

Does batch input ignore files already read?

Jason — Thu, 05 May 2011 17:49:22 GMT

I have a file in a folder that is being monitored by Splunk. Its contents have been indexed. If I move that file into a batch input (read-and-delete), will the file be re-indexed, or will Splunk know it has already indexed this data already?

(The specific scenario is I am changing a folder from monitor:// to batch:// and need to know if I need to remove all the files first to avoid data duplication in Splunk.)

Re: Does batch input ignore files already read?

jbsplunk — Thu, 05 May 2011 18:05:33 GMT

so long as the policy is set to move to the sinkhole, Splunk should eat the file again.
The input could look something like this:

[batch:///some/path/some_file]
move_policy = sinkhole

without the move_policy = sinkhole setting, it won't load the files destructively and will keep track of them.

Hope this helps!

Re: Does batch input ignore files already read?

Jason — Thu, 05 May 2011 18:14:57 GMT

Confirmed - Splunk does no checking and will re-index the file.