Improve performance of processing of a considerable number of batch input files

[fortran01]

It appears the batch processor during each iteration only deletes the log files (lined up in the queue for each iteration) right after it completes opening/seeking and forwarding. This requires us to throttle our log generation facility considerably. Can you recommend best practices so we can cope our log generation facility. TIA.

[gkanapathy · Accepted solution]

Well yes. Until the data has been indexed, it can't be deleted, and must be stored somewhere. While Splunk has internal queues that can hold some amount of data, there's no advantage to using those rather than simply leaving them on the file system in the batch directory. I don't see why you feel you need to throttle your log generation, or what you think the forwarder would do if you're generating data faster than it can be sent.

Of course you can increase throughput by raising the forwarder max thruput soft limit, and if that isn't sufficient, then you must install additional indexing capacity.

[fortran01]

Already set to 0. Does organizing into sub-directories factor?

[gkanapathy]

the setting is [thruput] maxKBps in limits.conf

[fortran01]

We increased indexing capacity. It appears to work.

Any hints how to raise forwarder max thruput soft limit?