Getting Data In

Improve performance of processing of a considerable number of batch input files

fortran01
Explorer

It appears the batch processor during each iteration only deletes the log files (lined up in the queue for each iteration) right after it completes opening/seeking and forwarding. This requires us to throttle our log generation facility considerably. Can you recommend best practices so we can cope our log generation facility. TIA.

Tags (1)
1 Solution

gkanapathy
Splunk Employee
Splunk Employee

Well yes. Until the data has been indexed, it can't be deleted, and must be stored somewhere. While Splunk has internal queues that can hold some amount of data, there's no advantage to using those rather than simply leaving them on the file system in the batch directory. I don't see why you feel you need to throttle your log generation, or what you think the forwarder would do if you're generating data faster than it can be sent.

Of course you can increase throughput by raising the forwarder max thruput soft limit, and if that isn't sufficient, then you must install additional indexing capacity.

View solution in original post

gkanapathy
Splunk Employee
Splunk Employee

Well yes. Until the data has been indexed, it can't be deleted, and must be stored somewhere. While Splunk has internal queues that can hold some amount of data, there's no advantage to using those rather than simply leaving them on the file system in the batch directory. I don't see why you feel you need to throttle your log generation, or what you think the forwarder would do if you're generating data faster than it can be sent.

Of course you can increase throughput by raising the forwarder max thruput soft limit, and if that isn't sufficient, then you must install additional indexing capacity.

fortran01
Explorer

Already set to 0. Does organizing into sub-directories factor?

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

the setting is [thruput] maxKBps in limits.conf

0 Karma

fortran01
Explorer

We increased indexing capacity. It appears to work.

Any hints how to raise forwarder max thruput soft limit?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...