Getting Data In

Could you provide more monitoring detail on a directory monitored via batch/sinkhole ?

Chris_R_
Splunk Employee
Splunk Employee

I have 10's of thousands of files(tarballs) i want to monitor via batch/sinkhole.

[batch:///var/log/archived_files]
index = someindex
move_policy = sinkhole
sourcetype = unique_sourcetype

With a batch/sinkhole input, how frequently is the directory checked?
Do I need to move the file into the directory atomicly (i.e. a mv from the local filesystem), or can splunk figure out when the file is not open for writing by anyone?

Tags (2)
1 Solution

amrit
Splunk Employee
Splunk Employee

The answer for how often directories are monitored is basically "as often as we can". The exact timing varies with the frequency of changes to the directory.

Files moved into the sinkhole should definitely be moved atomically - we will process the file as soon as we see it. Post 4.1.2, if it's an archive file that we recognize, we'll make sure it hasn't been written to for at least 10 seconds before eating the file.

View solution in original post

amrit
Splunk Employee
Splunk Employee

The answer for how often directories are monitored is basically "as often as we can". The exact timing varies with the frequency of changes to the directory.

Files moved into the sinkhole should definitely be moved atomically - we will process the file as soon as we see it. Post 4.1.2, if it's an archive file that we recognize, we'll make sure it hasn't been written to for at least 10 seconds before eating the file.

Lowell
Super Champion

I'm pretty sure that atomic moves are always a good thing. I know splunk does renames when moving files into $SPLUNK_HOME/var/spool/splunk when writing summary indexing .stash files, for example.

0 Karma
Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...