https://community.splunk.com/t5/Getting-Data-In/How-to-make-Splunk-read-my-files-faster/m-p/122160#M25296 <P>Note that the min_batch_size_bytes is NOT actually bytes, but MB (see the limits.conf docs). So 2GB should be 2048, not 2147000000.</P> Tue, 29 Sep 2020 14:54:53 GMT vliggio 2020-09-29T14:54:53Z https://community.splunk.com/t5/Getting-Data-In/How-to-make-Splunk-read-my-files-faster/m-p/122157#M25293 <P>I have a folder of 100 1GB files on a forwarder that I need to get into Splunk ASAP via a <CODE>monitor://</CODE>. One forwarder, distributing to 10 indexers. As such, I need to read as many of these files as I can at once.</P> <P>The forwarder has unlimited upload, and I have raised the threshold for using the BatchReader - but STILL it reads files one at a time - how do I fix this?</P> <PRE><CODE># limits.conf # Raise limit after which Splunk will use single-threaded BatchReader to 2GB min_batch_size_bytes = 2147000000 # Unlimit output [thruput] maxKBps = 0 </CODE></PRE> Thu, 11 Sep 2014 23:33:17 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-make-Splunk-read-my-files-faster/m-p/122157#M25293 Jason 2014-09-11T23:33:17Z https://community.splunk.com/t5/Getting-Data-In/How-to-make-Splunk-read-my-files-faster/m-p/122158#M25294 <P>More a work around than a fix: split the files into 10 directories, run multiple instances of the UF each <CODE>monitoring</CODE> one directory and distributing to the 10 indexers.</P> Fri, 12 Sep 2014 06:01:18 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-make-Splunk-read-my-files-faster/m-p/122158#M25294 MuS 2014-09-12T06:01:18Z https://community.splunk.com/t5/Getting-Data-In/How-to-make-Splunk-read-my-files-faster/m-p/122159#M25295 <P><CODE>forceTimebasedAutoLB</CODE> should make better use of your indexing tier in this case. It might not help with the "read more than one file at a time" bit, but it'll spread the load out to the indexer tier more uniformly, taking advantage of input queues and the like. Your current way is probably ending up with "one file to that indexer, one file to the second indexer", etc. switching only on EOF.</P> Fri, 12 Sep 2014 20:58:34 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-make-Splunk-read-my-files-faster/m-p/122159#M25295 sowings 2014-09-12T20:58:34Z https://community.splunk.com/t5/Getting-Data-In/How-to-make-Splunk-read-my-files-faster/m-p/122160#M25296 <P>Note that the min_batch_size_bytes is NOT actually bytes, but MB (see the limits.conf docs). So 2GB should be 2048, not 2147000000.</P> Tue, 29 Sep 2020 14:54:53 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-make-Splunk-read-my-files-faster/m-p/122160#M25296 vliggio 2020-09-29T14:54:53Z https://community.splunk.com/t5/Getting-Data-In/How-to-make-Splunk-read-my-files-faster/m-p/122161#M25297 <P>I believe this might have been changed to MB in 6.6.x (just comparing the min_batch_size_bytes text in limits.conf spec). Any references to this, that I've seen, have always been in bytes.</P> Tue, 29 Sep 2020 15:14:33 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-make-Splunk-read-my-files-faster/m-p/122161#M25297 hays2 2020-09-29T15:14:33Z