Could you provide more monitoring detail on a directory monitored via batch/sinkhole ?

[Chris_R_]

I have 10's of thousands of files(tarballs) i want to monitor via batch/sinkhole.

[batch:///var/log/archived_files]
index = someindex
move_policy = sinkhole
sourcetype = unique_sourcetype

With a batch/sinkhole input, how frequently is the directory checked?
Do I need to move the file into the directory atomicly (i.e. a mv from the local filesystem), or can splunk figure out when the file is not open for writing by anyone?

[amrit · Accepted solution]

The answer for how often directories are monitored is basically "as often as we can". The exact timing varies with the frequency of changes to the directory.

Files moved into the sinkhole should definitely be moved atomically - we will process the file as soon as we see it. Post 4.1.2, if it's an archive file that we recognize, we'll make sure it hasn't been written to for at least 10 seconds before eating the file.

[Lowell]

I'm pretty sure that atomic moves are always a good thing. I know splunk does renames when moving files into $SPLUNK_HOME/var/spool/splunk when writing summary indexing .stash files, for example.