Getting Data In

Improve performance of processing of a considerable number of batch input files

fortran01
Explorer

It appears the batch processor during each iteration only deletes the log files (lined up in the queue for each iteration) right after it completes opening/seeking and forwarding. This requires us to throttle our log generation facility considerably. Can you recommend best practices so we can cope our log generation facility. TIA.

Tags (1)
1 Solution

gkanapathy
Splunk Employee
Splunk Employee

Well yes. Until the data has been indexed, it can't be deleted, and must be stored somewhere. While Splunk has internal queues that can hold some amount of data, there's no advantage to using those rather than simply leaving them on the file system in the batch directory. I don't see why you feel you need to throttle your log generation, or what you think the forwarder would do if you're generating data faster than it can be sent.

Of course you can increase throughput by raising the forwarder max thruput soft limit, and if that isn't sufficient, then you must install additional indexing capacity.

View solution in original post

gkanapathy
Splunk Employee
Splunk Employee

Well yes. Until the data has been indexed, it can't be deleted, and must be stored somewhere. While Splunk has internal queues that can hold some amount of data, there's no advantage to using those rather than simply leaving them on the file system in the batch directory. I don't see why you feel you need to throttle your log generation, or what you think the forwarder would do if you're generating data faster than it can be sent.

Of course you can increase throughput by raising the forwarder max thruput soft limit, and if that isn't sufficient, then you must install additional indexing capacity.

fortran01
Explorer

Already set to 0. Does organizing into sub-directories factor?

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

the setting is [thruput] maxKBps in limits.conf

0 Karma

fortran01
Explorer

We increased indexing capacity. It appears to work.

Any hints how to raise forwarder max thruput soft limit?

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...