Parallel data monitor/transmission - inputs.conf precedence

[rgaleone1]

Splunk:

Indexer <- series of tubes -> Forwarder

App:

fwdtosplunk/default/inputs.conf
[monitor:///path1/]
[monitor:///path2/]

Question:

Both path1 are path2 are large directories. Will the Forwarder need to completely send all data in path1 before beginning to sending data from path2?

[kristian_kolb]

The TailingProcessor, which is 'responsible' for checking which files have been updated (or are unread), used to be a single-threaded process in v5 (don't know if that has changed). Unless I'm much mistaken, that probably means that it would handle the input files sequentially.

However, if not all of those files are being updated (i.e. it's an archive of ooold files), you would only see this problem the first time you're indexing the files. Also, if it is an archive of old files that you DON'T want to be indexed, you could set the ignoreOlderThan parameter in inputs.conf or move the old files away to some other directory (don't put them in subdirectory, unless you also set recurse=false for that input. 🙂

Hope this helps a little,

K