Getting Data In
Highlighted

Could you provide more monitoring detail on a directory monitored via batch/sinkhole ?

Communicator

I have 10's of thousands of files(tarballs) i want to monitor via batch/sinkhole.

[batch:///var/log/archived_files]
index = someindex
move_policy = sinkhole
sourcetype = unique_sourcetype

With a batch/sinkhole input, how frequently is the directory checked?
Do I need to move the file into the directory atomicly (i.e. a mv from the local filesystem), or can splunk figure out when the file is not open for writing by anyone?

Tags (2)
Highlighted

Re: Could you provide more monitoring detail on a directory monitored via batch/sinkhole ?

Super Champion

I'm pretty sure that atomic moves are always a good thing. I know splunk does renames when moving files into $SPLUNK_HOME/var/spool/splunk when writing summary indexing .stash files, for example.

0 Karma
Highlighted

Re: Could you provide more monitoring detail on a directory monitored via batch/sinkhole ?

Contributor

The answer for how often directories are monitored is basically "as often as we can". The exact timing varies with the frequency of changes to the directory.

Files moved into the sinkhole should definitely be moved atomically - we will process the file as soon as we see it. Post 4.1.2, if it's an archive file that we recognize, we'll make sure it hasn't been written to for at least 10 seconds before eating the file.

View solution in original post

Speak Up for Splunk Careers!

We want to better understand the impact Splunk experience and expertise has has on individuals' careers, and help highlight the growing demand for Splunk skills.