I have 10's of thousands of files(tarballs) i want to monitor via batch/sinkhole.
index = someindex
move_policy = sinkhole
sourcetype = unique_sourcetype
With a batch/sinkhole input, how frequently is the directory checked?
Do I need to move the file into the directory atomicly (i.e. a mv from the local
filesystem), or can splunk figure out when the file is not open for writing by
The answer for how often directories are monitored is basically "as often as we can". The exact timing varies with the frequency of changes to the directory.
Files moved into the sinkhole should definitely be moved atomically - we will process the file as soon as we see it. Post 4.1.2, if it's an archive file that we recognize, we'll make sure it hasn't been written to for at least 10 seconds before eating the file.