Getting Data In

When is the BatchReader used and when is the TailingProcessor used?


I have a UniversalForwarder that is exporting hourly data to a monitored folder. Sometimes there are a lot of records (50,000+) and sometimes there is only a few thousand. I have noticed that when the file is bigger that the BatchReader appears to process the file and when the file is smaller the TailingProcessor processes the file.

Is there a limit to this that can be set?

0 Karma

Splunk Employee
Splunk Employee

The batch reader is used when the file is over 20 mb in size. Otherwise, the regular tailing processor queue is used. The batch reader only processes one file at a time, while the tailing processor can do many. The limit exists to prevent a bunch of large files for using up all slots and starving out new smaller files.

The threshold can be changed in limits.conf:

min_batch_size_bytes = 10485760

to set it to 10 MB, for example.


Is this documented somewhere in Splunk official documentation ?

Get Updates on the Splunk Community!

Build Scalable Security While Moving to Cloud - Guide From Clayton Homes

 Clayton Homes faced the increased challenge of strengthening their security posture as they went through ...

Mission Control | Explore the latest release of Splunk Mission Control (2.3)

We’re happy to announce the release of Mission Control 2.3 which includes several new and exciting features ...

Cloud Platform | Migrating your Splunk Cloud deployment to Python 3.7

Python 2.7, the last release of Python 2, reached End of Life back on January 1, 2020. As part of our larger ...