Getting Data In

What is the max file size that a universal forwarder can input via a batch stanza?

Path Finder

Splunk universal forwarder inputs.conf batch stanza is attempting to read CSV files that range in size from a 10MB to 2GB.
On the forwarder the splunkd.log shows "Stale file handle" and "CRC calculation" related warnings and errors on the larger files

i.e. 800MB and 1.4GB.
The files are not indexed and then they are deleted.

Are there hard or configured file size limits?
and/or What might cause these issues other than file size?

0 Karma


HI mdwecht-
File size should not really matter at all. It's all about throughput and events (IMHO).
Do your CSV files have multiple events or are they seen as one giant event? You might have to experiment with smaller chunks fo the same data in another CSV file to see how the events are appearing in Splunk.

For "Speed" - You may want to adjust this setting:
maxKBps =
in limits.conf file. There should be a default on on the forwarder:


But, make sure to only alter the ../local directory. If the limits.conf file is not there, just create one with the stanza I specified.
Ref: link text

For the CRC Calc errors, are you altering the file in any manner? If the file is altered before Splunk can finish reading it through, that can be an issue:
Reference: link text

Hope this helps,

0 Karma

Path Finder

The CSV files have an event per line, when the smaller files of same sourcetype are read and indexed the events look fine in splunk, universal forwarders out of the box are throughput limited so we always set maxKBps = 0. The actual warnings/errors I get are "WARN FileInputTracker - Error reading CRC: Stale file handle" and/or "ERROR WatchedFile - Error reading file". The files are not in motion when Splunk is reading them. They are delivered to a folder via nifi safe write i.e. COPIED into the folder named with a preceding "." and then MOVED to the name the Splunk inputs.conf batch stanza is looking for without the preceding ".".

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In the last month, the Splunk Threat Research Team (STRT) has had 2 releases of new security content via the ...

Announcing the 1st Round Champion’s Tribute Winners of the Great Resilience Quest

We are happy to announce the 20 lucky questers who are selected to be the first round of Champion's Tribute ...

We’ve Got Education Validation!

Are you feeling it? All the career-boosting benefits of up-skilling with Splunk? It’s not just a feeling, it's ...