After reviewing splund.log, metrics.log in several attempts and adding check on storage etc. on splunk servers, we have received one pattern.
The only logs getting delayed by forwarder are the one's generated by multiple threads in an application. We have several single threaded applications that generate huge logs, but they don't get delayed during indexing.
While the one app running parallel jobs in multiple threads and generating relatively less logs, still gets delayed in indexing.
Is there any throttle limit that can be adjusted for allowing more logs to get forwarded to indexers per second?
