How to make Splunk read my files faster?

[Jason]

I have a folder of 100 1GB files on a forwarder that I need to get into Splunk ASAP via a monitor://. One forwarder, distributing to 10 indexers. As such, I need to read as many of these files as I can at once.

The forwarder has unlimited upload, and I have raised the threshold for using the BatchReader - but STILL it reads files one at a time - how do I fix this?

# limits.conf
# Raise limit after which Splunk will use single-threaded BatchReader to 2GB
min_batch_size_bytes = 2147000000

# Unlimit output
[thruput]
maxKBps = 0

[sowings]

forceTimebasedAutoLB should make better use of your indexing tier in this case. It might not help with the "read more than one file at a time" bit, but it'll spread the load out to the indexer tier more uniformly, taking advantage of input queues and the like. Your current way is probably ending up with "one file to that indexer, one file to the second indexer", etc. switching only on EOF.

[MuS]

More a work around than a fix: split the files into 10 directories, run multiple instances of the UF each monitoring one directory and distributing to the 10 indexers.

[vliggio]

Note that the min_batch_size_bytes is NOT actually bytes, but MB (see the limits.conf docs). So 2GB should be 2048, not 2147000000.

[hays2]

I believe this might have been changed to MB in 6.6.x (just comparing the min_batch_size_bytes text in limits.conf spec). Any references to this, that I've seen, have always been in bytes.