topic Re: Improve performance of processing of a considerable number of batch input files in Getting Data In

Improve performance of processing of a considerable number of batch input files

fortran01 — Mon, 04 Jun 2012 20:42:23 GMT

It appears the batch processor during each iteration only deletes the log files (lined up in the queue for each iteration) right after it completes opening/seeking and forwarding. This requires us to throttle our log generation facility considerably. Can you recommend best practices so we can cope our log generation facility. TIA.

Re: Improve performance of processing of a considerable number of batch input files

gkanapathy — Mon, 04 Jun 2012 20:58:55 GMT

Well yes. Until the data has been indexed, it can't be deleted, and must be stored somewhere. While Splunk has internal queues that can hold some amount of data, there's no advantage to using those rather than simply leaving them on the file system in the batch directory. I don't see why you feel you need to throttle your log generation, or what you think the forwarder would do if you're generating data faster than it can be sent.

Of course you can increase throughput by raising the forwarder max thruput soft limit, and if that isn't sufficient, then you must install additional indexing capacity.

Re: Improve performance of processing of a considerable number of batch input files

fortran01 — Thu, 07 Jun 2012 22:40:22 GMT

We increased indexing capacity. It appears to work.

Any hints how to raise forwarder max thruput soft limit?

Re: Improve performance of processing of a considerable number of batch input files

gkanapathy — Thu, 07 Jun 2012 22:54:40 GMT

the setting is [thruput] maxKBps in limits.conf

Re: Improve performance of processing of a considerable number of batch input files

fortran01 — Fri, 08 Jun 2012 03:17:14 GMT

Already set to 0. Does organizing into sub-directories factor?