Getting Data In

Parallel data monitor/transmission - inputs.conf precedence

rgaleone1
Path Finder

Splunk:

Indexer <- series of tubes -> Forwarder

App:

fwdtosplunk/default/inputs.conf
[monitor:///path1/]
[monitor:///path2/]

Question:

Both path1 are path2 are large directories. Will the Forwarder need to completely send all data in path1 before beginning to sending data from path2?
0 Karma

kristian_kolb
Ultra Champion

The TailingProcessor, which is 'responsible' for checking which files have been updated (or are unread), used to be a single-threaded process in v5 (don't know if that has changed). Unless I'm much mistaken, that probably means that it would handle the input files sequentially.

However, if not all of those files are being updated (i.e. it's an archive of ooold files), you would only see this problem the first time you're indexing the files. Also, if it is an archive of old files that you DON'T want to be indexed, you could set the ignoreOlderThan parameter in inputs.conf or move the old files away to some other directory (don't put them in subdirectory, unless you also set recurse=false for that input. 🙂

Hope this helps a little,

K

Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...