Getting Data In

Could you provide more monitoring detail on a directory monitored via batch/sinkhole ?

Chris_R_
Splunk Employee
Splunk Employee

I have 10's of thousands of files(tarballs) i want to monitor via batch/sinkhole.

[batch:///var/log/archived_files]
index = someindex
move_policy = sinkhole
sourcetype = unique_sourcetype

With a batch/sinkhole input, how frequently is the directory checked?
Do I need to move the file into the directory atomicly (i.e. a mv from the local filesystem), or can splunk figure out when the file is not open for writing by anyone?

Tags (2)
1 Solution

amrit
Splunk Employee
Splunk Employee

The answer for how often directories are monitored is basically "as often as we can". The exact timing varies with the frequency of changes to the directory.

Files moved into the sinkhole should definitely be moved atomically - we will process the file as soon as we see it. Post 4.1.2, if it's an archive file that we recognize, we'll make sure it hasn't been written to for at least 10 seconds before eating the file.

View solution in original post

amrit
Splunk Employee
Splunk Employee

The answer for how often directories are monitored is basically "as often as we can". The exact timing varies with the frequency of changes to the directory.

Files moved into the sinkhole should definitely be moved atomically - we will process the file as soon as we see it. Post 4.1.2, if it's an archive file that we recognize, we'll make sure it hasn't been written to for at least 10 seconds before eating the file.

View solution in original post

Lowell
Super Champion

I'm pretty sure that atomic moves are always a good thing. I know splunk does renames when moving files into $SPLUNK_HOME/var/spool/splunk when writing summary indexing .stash files, for example.

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!