https://community.splunk.com/t5/Getting-Data-In/When-is-the-BatchReader-used-and-when-is-the-TailingProcessor/m-p/131849#M27122 <P>I have a UniversalForwarder that is exporting hourly data to a monitored folder. Sometimes there are a lot of records (50,000+) and sometimes there is only a few thousand. I have noticed that when the file is bigger that the BatchReader appears to process the file and when the file is smaller the TailingProcessor processes the file.</P> <P>Is there a limit to this that can be set?</P> Thu, 07 Nov 2013 04:27:18 GMT JeremyHagan 2013-11-07T04:27:18Z https://community.splunk.com/t5/Getting-Data-In/When-is-the-BatchReader-used-and-when-is-the-TailingProcessor/m-p/131849#M27122 <P>I have a UniversalForwarder that is exporting hourly data to a monitored folder. Sometimes there are a lot of records (50,000+) and sometimes there is only a few thousand. I have noticed that when the file is bigger that the BatchReader appears to process the file and when the file is smaller the TailingProcessor processes the file.</P> <P>Is there a limit to this that can be set?</P> Thu, 07 Nov 2013 04:27:18 GMT https://community.splunk.com/t5/Getting-Data-In/When-is-the-BatchReader-used-and-when-is-the-TailingProcessor/m-p/131849#M27122 JeremyHagan 2013-11-07T04:27:18Z https://community.splunk.com/t5/Getting-Data-In/When-is-the-BatchReader-used-and-when-is-the-TailingProcessor/m-p/131850#M27123 <P>The batch reader is used when the file is over 20 mb in size. Otherwise, the regular tailing processor queue is used. The batch reader only processes one file at a time, while the tailing processor can do many. The limit exists to prevent a bunch of large files for using up all slots and starving out new smaller files. </P> <P>The threshold can be changed in limits.conf:</P> <PRE><CODE>min_batch_size_bytes = 10485760 </CODE></PRE> <P>to set it to 10 MB, for example.</P> Thu, 07 Nov 2013 07:33:53 GMT https://community.splunk.com/t5/Getting-Data-In/When-is-the-BatchReader-used-and-when-is-the-TailingProcessor/m-p/131850#M27123 gkanapathy 2013-11-07T07:33:53Z https://community.splunk.com/t5/Getting-Data-In/When-is-the-BatchReader-used-and-when-is-the-TailingProcessor/m-p/561156#M92705 <P>Is this documented somewhere in Splunk official documentation ?</P> Wed, 28 Jul 2021 08:45:33 GMT https://community.splunk.com/t5/Getting-Data-In/When-is-the-BatchReader-used-and-when-is-the-TailingProcessor/m-p/561156#M92705 dm1 2021-07-28T08:45:33Z